Machine learning systems and methods for assessment, healing prediction, and treatment of wounds

ABSTRACT

Machine learning systems and methods are disclosed for prediction of wound healing, such as for diabetic foot ulcers or other wounds, and for assessment implementations such as segmentation of images into wound regions and non-wound regions. Systems for assessing or predicting wound healing can include a light detection element configured to collect light of at least a first wavelength reflected from a tissue region including a wound, and one or more processors configured to generate an image based on a signal from the light detection element having pixels depicting the tissue region, determine reflectance intensity values for at least a subset of the pixels, determine one or more quantitative features of the subset of the plurality of pixels based on the reflectance intensity values, and generate a predicted or assessed healing parameter associated with the wound over a predetermined time interval.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/013,336, filed Sep. 4, 2020, entitled “MACHINE LEARNING SYSTEMS ANDMETHODS FOR ASSESSMENT, HEALING PREDICTION, AND TREATMENT OF WOUNDS,”which is a continuation of Ser. No. 16/738,911, filed Jan. 9, 2020,entitled “MACHINE LEARNING SYSTEMS AND METHODS FOR ASSESSMENT, HEALINGPREDICTION, AND TREATMENT OF WOUNDS,” which is a continuation ofPCT/US2019/065820, filed Dec. 11, 2019, entitled “MACHINE LEARNINGSYSTEMS AND TECHNIQUES FOR ASSESSMENT, HEALING PREDICTION, AND TREATMENTOF WOUNDS,” which claims the benefit of U.S. Provisional ApplicationSer. No. 62/780,854, filed Dec. 17, 2018, entitled “PREDICTION OFDIABETIC FOOT ULCER HEALING UPON INITIAL VISIT USING ARTIFICIALINTELLIGENCE,” U.S. Provisional Application Ser. No. 62/780,121, filedDec. 14, 2018, entitled “SYSTEM AND METHOD FOR HIGH PRECISIONMULTI-APERTURE SPECTRAL IMAGING,” and U.S. Provisional Application Ser.No. 62/818,375, filed Mar. 14, 2019, entitled “SYSTEM AND METHOD FORHIGH PRECISION MULTI-APERTURE SPECTRAL IMAGING,” all of which are herebyexpressly incorporated by reference in their entirety and for allpurposes.

STATEMENT REGARDING FEDERALLY SPONSORED R&D

Some of the work described in this disclosure was made with UnitedStates Government support under Contract No. HHS0100201300022C, awardedby the Biomedical Advanced Research and Development Authority (BARDA),within the Office of the Assistant Secretary for Preparedness andResponse in the U.S. Department of Health and Human Services. Some ofthe work described in this disclosure was made with United Governmentsupport under Contract Nos. W81XWH-17-C-0170 and/or W81XWH-18-C-0114,awarded by the U.S. Defense Health Agency (DHA). The United StatesGovernment may have certain rights in this invention.

TECHNICAL FIELD

The systems and methods disclosed herein are directed to medicalimaging, and, more particularly, to wound assessment, healingprediction, and treatment using machine learning techniques.

BACKGROUND

Optical imaging is an emerging technology with potential for improvingdisease prevention, diagnosis, and treatment at the scene of anemergency, in the medical office, at the bedside, or in the operatingroom. Optical imaging technologies can noninvasively differentiate amongtissues, and between native tissues and tissue labeled with eitherendogenous or exogenous contrast media, measuring their different photonabsorption or scattering profiles at different wavelengths. Such photonabsorption and scattering differences offers potential for providingspecific tissue contrasts, and enables studying functional and molecularlevel activities that are the basis for health and disease.

The electromagnetic spectrum is the range of wavelengths or frequenciesover which electromagnetic radiation (e.g., light) extends. In orderfrom longer wavelengths to shorter wavelengths, the electromagneticspectrum includes radio waves, microwaves, infrared (IR) light, visiblelight (that is, light that is detectable by the structures of the humaneye), ultraviolet (UV) light, x-rays, and gamma rays. Spectral imagingrefers to a branch of spectroscopy and photography in which somespectral information or a complete spectrum is collected at locations inan image plane. Some spectral imaging systems can capture one or morespectral bands. Multispectral imaging systems can capture multiplespectral bands (on the order of a dozen or less and typically atdiscrete spectral regions), for which spectral band measurements arecollected at each pixel, and can refer to bandwidths of about tens ofnanometers per spectral channel. Hyperspectral imaging systems measure agreater number of spectral bands, for example as many as 200 or more,with some providing a continuous sampling of narrow bands (e.g.,spectral bandwidths on the order of nanometers or less) along a portionof the electromagnetic spectrum.

SUMMARY

Aspects of the technology described herein relate to devices and methodsthat can be used to assess and/or classify tissue regions at or near awound using non-contact, non-invasive, and non-radiation opticalimaging. Such devices and methods may, for example, identify tissueregions corresponding to different tissue health classificationsrelating to wounds and/or determine predicted healing parameters for awound or a portion thereof, and can output a visual representation ofthe identified regions and/or parameters for use by a clinician indetermining a wound healing prognosis or selecting an appropriate woundcare therapy or both. In some embodiments, the devices and methods ofthe present technology can provide such classification and/or predictionbased on imaging at a single wavelength or at a plurality ofwavelengths. There has been a long felt need for non-invasive imagingtechniques that can provide physicians with information forquantitatively predicting healing for wounds or portions thereof.

In one aspect, a system for assessing or predicting wound healingcomprises at least one light detection element configured to collectlight of at least a first wavelength after being reflected from a tissueregion comprising a wound, and one or more processors in communicationwith the at least one light detection element. The one or moreprocessors are configured to receive a signal from the at least onelight detection element, the signal representing light of the firstwavelength reflected from the tissue region; generate, based on thesignal, an image having a plurality of pixels depicting the tissueregion; determine, based on the signal, a reflectance intensity value atthe first wavelength for each pixel of at least a subset of theplurality of pixels; determine one or more quantitative features of thesubset of the plurality of pixels based on the reflectance intensityvalues of each pixel of the subset; and generate, using one or moremachine learning algorithms, at least one scalar value based on the oneor more quantitative features of the subset of the plurality of pixels,the at least one scalar value corresponding to a predicted or assessedhealing parameter over a predetermined time interval.

In some embodiments, the wound is a diabetic foot ulcer. In someembodiments, the predicted healing parameter is a predicted amount ofhealing of the wound. In some embodiments, the predicted healingparameter is a predicted percent area reduction of the wound. In someembodiments, the at least one scalar value comprises a plurality ofscalar values, each scalar value of the plurality of scalar valuescorresponding to a probability of healing of an individual pixel of thesubset or of a subgroup of individual pixels of the subset. In someembodiments, the one or more processors are further configured to outputa visual representation of the plurality of scalar values for display toa user. In some embodiments, the visual representation comprises theimage having each pixel of the subset displayed with a particular visualrepresentation selected based on the probability of healingcorresponding to the pixel, wherein pixels associated with differentprobabilities of healing are displayed in different visualrepresentations. In some embodiments, the one or more machine learningalgorithms comprise a SegNet pre-trained using a wound, burn, or ulcerimage database. In some embodiments, the wound image database comprisesa diabetic foot ulcer image database. In some embodiments, the woundimage database comprises a burn image database. In some embodiments, thepredetermined time interval is 30 days. In some embodiments, the one ormore processors are further configured to identify at least one patienthealth metric value corresponding to a patient having the tissue region,and wherein the at least one scalar value is generated based on the oneor more quantitative features of the subset of the plurality of pixelsand on the at least one patient health metric value. In someembodiments, the at least one patient health metric value comprises atleast one variable selected from the group consisting of demographicvariables, diabetic foot ulcer history variables, compliance variables,endocrine variables, cardiovascular variables, musculoskeletalvariables, nutrition variables, infectious disease variables, renalvariables, obstetrics or gynecology variables, drug use variables, otherdisease variables, or laboratory values. In some embodiments, the atleast one patient health metric value comprises one or more clinicalfeatures. In some embodiments, the one or more clinical featurescomprise at least one feature selected from the group consisting of anage of the patient, a level of chronic kidney disease of the patient, alength of the wound on a day when the image is generated, and a width ofthe wound on the day when the image is generated. In some embodiments,the first wavelength is within the range of 420 nm±20 nm, 525 nm±35 nm,581 nm±20 nm, 620 nm±20 nm, 660 nm±20 nm, 726 nm±41 nm, 820 nm±20 nm, or855 nm±30 nm. In some embodiments, the first wavelength is within therange of 620 nm±20 nm, 660 nm±20 nm, or 420 nm±20 nm. In someembodiments, the one or more machine learning algorithms comprise arandom forest ensemble. In some embodiments, the first wavelength iswithin the range of 726 nm±41 nm, 855 nm±30 nm, 525 nm±35 nm, 581 nm±20nm, or 820 nm±20 nm. In some embodiments, the one or more machinelearning algorithms comprise an ensemble of classifiers. In someembodiments, the system further comprises an optical bandpass filterconfigured to pass light of at least the first wavelength. In someembodiments, the one or more processors are further configured toautomatically segment the plurality of pixels of the image into woundpixels and non-wound pixels, and select the subset of the plurality ofpixels to comprise the wound pixels. In some embodiments, the one ormore processors are further configured to automatically segment thenon-wound pixels into callus pixels and background pixels. In someembodiments, the one or more processors are further configured toautomatically segment the non-wound pixels into callus pixels, normalskin pixels, and background pixels. In some embodiments, the one or moreprocessors automatically segment the plurality of pixels using asegmentation algorithm comprising a convolutional neural network. Insome embodiments, the segmentation algorithm is at least one of a U-Netcomprising a plurality of convolutional layers and a SegNet comprising aplurality of convolutional layers. In some embodiments, the one or morequantitative features of the subset of the plurality of pixels compriseone or more aggregate quantitative features of the plurality of pixels.In some embodiments, the one or more aggregate quantitative features ofthe subset of the plurality of pixels are selected from the groupconsisting of a mean of the reflectance intensity values of the pixelsof the subset, a standard deviation of the reflectance intensity valuesof the pixels of the subset, and a median reflectance intensity value ofthe pixels of the subset. In some embodiments, the one or moreprocessors are further configured to individually apply a plurality offilter kernels to the image by convolution to generate a plurality ofimage transformations; construct a 3D matrix from the plurality of imagetransformations; and determine one or more quantitative features of the3D matrix, wherein the at least one scalar value is generated based onthe one or more quantitative features of the subset of the plurality ofpixels and on the one or more quantitative features of the 3D matrix. Insome embodiments, the one or more quantitative features of the 3D matrixare selected from the group consisting of a mean of the values of the 3Dmatrix, a standard deviation of the values of the 3D matrix, a medianvalue of the 3D matrix, and a product of the mean and the median of the3D matrix. In some embodiments, the at least one scalar value isgenerated based on the mean of the reflectance intensity values of thepixels of the subset, the standard deviation of the reflectanceintensity values of the pixels of the subset, the median reflectanceintensity value of the pixels of the subset, the mean of the values ofthe 3D matrix, the standard deviation of the values of the 3D matrix,and the median value of the 3D matrix. In some embodiments, the at leastone light detection element is further configured to collect light of atleast a second wavelength after being reflected from the tissue region,and the one or more processors are further configured to receive asecond signal from the at least one light detection element, the secondsignal representing light of the second wavelength reflected from thetissue region; determine, based on the second signal, a reflectanceintensity value at the second wavelength for each pixel of at least thesubset of the plurality of pixels; and determine one or more additionalquantitative features of the subset of the plurality of pixels based onthe reflectance intensity values of each pixel at the second wavelength,wherein the at least one scalar value is generated based at least inpart on the one or more additional quantitative features of the subsetof the plurality of pixels.

In a second aspect, a system for wound assessment comprises at least onelight detection element configured to collect light of at least a firstwavelength after being reflected form a tissue region comprising awound, and one or more processors in communication with the at least onelight detection element. The one or more processors are configured toreceive a signal from the at least one light detection element, thesignal representing light of the first wavelength reflected from thetissue region; generate, based on the signal, an image having aplurality of pixels depicting the tissue region; determine, based on thesignal, a reflectance intensity value at the first wavelength for eachpixel of the plurality of pixels; and automatically segment, using amachine learning algorithm, individual pixels of the plurality of pixelsinto at least a first subset of the plurality of pixels comprising woundpixels and a second subset of the plurality of pixels comprisingnon-wound pixels, based on individual reflectance intensity values ofthe plurality of pixels.

In some embodiments, the one or more processors are further configuredto automatically segment the second subset of the plurality of pixelsinto at least two categories of non-wound pixels, the at least twocategories selected from the group consisting of callus pixels, normalskin pixels, and background pixels. In some embodiments, the machinelearning algorithm comprises a convolutional neural network. In someembodiments, the machine learning algorithm is at least one of a U-Netcomprising a plurality of convolutional layers and a SegNet comprising aplurality of convolutional layers. In some embodiments, the machinelearning algorithm is trained based on a dataset comprising a pluralityof segmented images of wounds, ulcers, or burns. In some embodiments,the wound is a diabetic foot ulcer. In some embodiments, the one or moreprocessors are further configured to output a visual representation ofthe segmented plurality of pixels for display to a user. In someembodiments, the visual representation comprises the image having eachpixel displayed with a particular visual representation selected basedon the segmentation of the pixel, wherein wound pixels and non-woundpixels are displayed in different visual representations.

In another aspect, a method of predicting wound healing using a systemfor assessing or predicting wound healing comprises illuminating thetissue region with light of at least the first wavelength such that thetissue region reflects at least a portion of the light to the at leastone light detection element, using the system to generate the at leastone scalar value, and determining the predicted healing parameter overthe predetermined time interval.

In some embodiments, illuminating the tissue region comprises activatingone or more light emitters configured to emit light of at least thefirst wavelength. In some embodiments, illuminating the tissue regioncomprises exposing the tissue region to ambient light. In someembodiments, determining the predicted healing parameter comprisesdetermining an expected percent area reduction of the wound over thepredetermined time interval. In some embodiments, the method furthercomprises measuring one or more dimensions of the wound after thepredetermined time interval has elapsed following the determination ofthe predicted amount of healing of the wound, determining an actualamount of healing of the wound over the predetermined time interval, andupdating at least one machine learning algorithm of the one or moremachine learning algorithms by providing at least the image and theactual amount of healing of the wound as training data. In someembodiments, the method further comprises selecting between a standardwound care therapy and an advanced wound care therapy based at least inpart on the predicted healing parameter. In some embodiments, selectingbetween the standard wound care therapy and the advanced wound caretherapy comprises, when the predicted healing parameter indicates thatthe wound, preferably a DFU, will heal or close by greater than 50% in30 days, indicating or applying one or more standard therapies selectedfrom the group consisting of optimization of nutritional status,debridement by any means to remove devitalized tissue, maintenance of aclean moist bed of granulation tissue with appropriate moist dressings,necessary therapy to resolve any infection that may be present,addressing any deficiencies in vascular perfusion to the extremity withthe DFU, offloading of pressure from the DFU, and appropriate glucosecontrol; and when the predicted healing parameter indicates that thewound, preferably a DFU, will not heal or close by greater than 50% in30 days, indicating or applying one or more advanced care therapiesselected from the group consisting of hyperbaric oxygen therapy,negative-pressure wound therapy, bioengineered skin substitutes,synthetic growth factors, extracellular matrix proteins, matrixmetalloproteinase modulators, and electrical stimulation therapy.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example of light incident on a filter atdifferent chief ray angles.

FIG. 1B is a graph illustrating example transmission efficienciesprovided by the filter of FIG. 1A for various chief ray angles.

FIG. 2A illustrates an example of a multispectral image datacube.

FIG. 2B illustrates examples of how certain multispectral imagingtechnologies generate the datacube of FIG. 2A.

FIG. 2C depicts an example snapshot imaging system that can generate thedatacube of FIG. 2A.

FIG. 3A depicts a schematic cross-sectional view of an optical design ofan example multi-aperture imaging system with curved multi-bandpassfilters, according to the present disclosure.

FIGS. 3B-3D depict example optical designs for optical components of onelight path of the multi-aperture imaging system of FIG. 3A.

FIGS. 4A-4E depict an embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B.

FIG. 5 depicts another embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B.

FIGS. 6A-6C depict another embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B.

FIGS. 7A-7B depict another embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B.

FIGS. 8A-8B depict another embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B.

FIGS. 9A-9C depict another embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B.

FIGS. 10A-10B depict another embodiment of a multispectralmulti-aperture imaging system, with an optical design as described withrespect to FIGS. 3A and 3B.

FIGS. 11A-11B depict an example set of wavebands that can be passed bythe filters of the multispectral multi-aperture imaging systems of FIGS.3A-10B.

FIG. 12 depicts a schematic block diagram of an imaging system that canbe used for the multispectral multi-aperture imaging systems of FIGS.3A-10B.

FIG. 13 is a flowchart of an example process for capturing image datausing the multispectral multi-aperture imaging systems of FIGS. 3A-10B.

FIG. 14 depicts a schematic block diagram of a workflow for processingimage data, for example image data captured using the process of FIG. 13and/or using the multispectral multi-aperture imaging systems of FIGS.3A-10B.

FIG. 15 graphically depicts disparity and disparity correction forprocessing image data, for example image data captured using the processof FIG. 13 and/or using the multispectral multi-aperture imaging systemsof FIGS. 3A-10B.

FIG. 16 graphically depicts a workflow for performing pixel-wiseclassification on multispectral image data, for example image datacaptured using the process of FIG. 13 , processed according to FIGS. 14and 15 , and/or using the multispectral multi-aperture imaging systemsof FIGS. 3A-10B.

FIG. 17 depicts a schematic block diagram of an example distributedcomputing system including the multispectral multi-aperture imagingsystems of FIGS. 3A-10B.

FIGS. 18A-18C illustrate an example handheld embodiment of amultispectral, multi-aperture imaging system.

FIGS. 19A and 19B illustrate an example handheld embodiment of amultispectral, multi-aperture imaging system.

FIGS. 20A and 20B illustrate an example multispectral, multi-apertureimaging system for a small USB 3.0 enclosed in a common camera housing.

FIG. 21 illustrates an example multispectral, multi-aperture imagingsystem including an additional illuminant for improved imageregistration.

FIG. 22 shows an example time progression of a healing diabetic footulcer (DFU) with corresponding area, volume, and debridementmeasurements.

FIG. 23 shows an example time progression of a non-healing DFU withcorresponding area, volume, and debridement measurements.

FIG. 24 schematically illustrates an example machine learning system forgenerating a healing prediction based on one or more images of a DFU.

FIG. 25 schematically illustrates an example machine learning system forgenerating a healing prediction based on one or more images of a DFU andone or more patient health metrics.

FIG. 26 illustrates an example set of wavelength bands used for spectraland/or multi-spectral imaging for image segmentation and/or generationof predicted healing parameters in accordance with the presenttechnology.

FIG. 27 is a histogram illustrating effects of the inclusion of clinicalvariables in example wound assessment methods of the present technology.

FIG. 28 schematically illustrates an example autoencoder in accordancewith the machine learning systems and methods of the present technology.

FIG. 29 schematically illustrates an example supervised machine learningalgorithm in accordance with the machine learning systems and methods ofthe present technology.

FIG. 30 schematically illustrates an example end-to-end machine learningalgorithm in accordance with the machine learning systems and methods ofthe present technology.

FIG. 31 is a bar graph illustrating the demonstrated accuracy of severalexample machine learning altorithms in accordance with the presenttechnology.

FIG. 32 is a bar graph illustrating the demonstrated accuracy of severalexample machine learning altorithms in accordance with the presenttechnology.

FIG. 33 schematically illustrates an example process of healingprediction and generation of a visual representation of a conditionalprobability mapping in accordance with the machine learning systems andmethods of the present technology.

FIG. 34 schematically illustrates an example conditional probabilitymapping algorithm including one or more feature-wise lineartransformation (FiLM) layers.

FIG. 35 illustrates the demonstrated accuracy of several imagesegmenation approaches for generating a conditional healing probabilitymap in accordance with the present technology.

FIG. 36 illustrates an example set of convolutional filter kernels usedin an example individual wavelength analysis method for healingprediction in accordance with the machine learning systems and methodsof the present technology.

FIG. 37 illustrates an example ground truth mask generated based on aDFU image for image segmentation in accordance with the machine learningsystems and methods of the present technology.

FIG. 38 illustrates the demonstrated accuracy of an example wound imagesegmentation algorithm in accordance with the machine learning systemsand methods of the present technology.

DETAILED DESCRIPTION

Approximately 15-25% of the 26 million Americans with diabetes willdevelop a diabetic foot ulcer (DFU). These wounds lead to a loss ofmobility, and lower quality of life. As many as 40% of those who developa DFU will develop a wound infection that increases the risk ofamputation and death. Mortality related to DFUs alone is as high as 5%during the first year and as high as 42% within five years. This isheightened by a high annual risk of major amputation (4.7%) and minoramputation (39.8%). Furthermore, the cost to treat one DFU annually isapproximately $22,000 to $44,000, and the overall burden to the U.S.healthcare system due to DFUs is in the range of $9 billion to $13billion per year.

It is generally accepted that DFUs with greater than 50% area reduction(PAR) after 30 days will heal by 12 weeks with standard of care therapy.However, using this metric requires four weeks of wound care before onecan determine if a more effective therapy (e.g., an advanced caretherapy) should be used. In a typical clinical approach to wound carefor non-urgent initial presentation, such as for a DFU, a patientreceives standard wound care therapy (e.g., correction of vascularproblems, optimization of nutrition, glucose control, debridement,dressings, and/or off-loading) for approximately 30 days following thepresentation and initial assessment of the wound. At approximately day30, the wound is assessed to determine if it is healing (e.g., percentarea reduction of greater than 50%). If the wound is not healingsufficiently, the treatment is supplemented with one or more advancedwound management therapies, which may include growth factors,bioengineered tissues, hyperbaric oxygen, negative pressure, amputation,recombinant human platelet-derived growth factor (e.g., Regranex™ Gel),bioengineered human dermal substitutes (e.g., Dermagraft™), and/orliving, bi-layered skin substitutes (e.g., Apligraf™). However,approximately 60% of DFUs fail to show sufficient healing after 30 daysof standard wound care therapy. In addition, approximately 40% of DFUswith early healing still fail to heal by 12 weeks, and median DFUhealing time has been estimated at 147 days, 188 days, and 237 days fortoe, midfoot, and heel ulcers, respectively.

DFUs that fail to achieve desirable healing after 30 days ofconventional or standard of care wound thereapy would benefit from theprovision of advanced wound care therapies as early as possible e.g.,during the initial 30 days of wound therapy. However, using conventionalassessment methods, physicians typically cannot accurately identify aDFU that will not respond to 30 days of standard wound care therapy.Many successful strategies that improve DFU therapy are available butare not prescribed until standard wound care therapy is ruled outempirically. Physiologic measurement devices have been used to attemptto diagnose the healing potential of a DFU, such as transcutaneousoxygen measurement, laser Doppler imaging, and indocyanine greenvideoangiography. However, these devices have suffered from inaccuracy,lack of useful data, lack of sensitivity, and prohibitively high cost,and thuse have not been suitable for widespread use in the assessment ofDFUs and other wounds. Clearly, an earlier and more accurate means ofpredicting DFU or other wound healing is important to quickly determinethe best therapy and reduce time to wound closure.

Generally described, the present technology provides non-invasive andpoint-of-care imaging devices capable of diagnosing the healingpotential of DFUs, burns, and other wounds. In various embodiments, thesystems and methods of the present technology can enable a clinician todetermine, at or shortly after the time of presentation or initialassessment, the healing potential of the wound. In some embodiments, thepresent technology can enable the determination of healing potential ofindividual sections of a wound, such as a DFU or burn. Based on thepredicted healing potential, a decision between standard and advancedwound care therapies can be made on or near day 0 of therapy, ratherthan being deferred until over 4 weeks from the initial presentation.Accordingly, the present technology may result in reduced healing timesand fewer amputations.

Example Spectral and Multi-Spectral Imaging Systems

Various spectral and multi-spectral imaging systems will now bedescribed, each of which may be used in accordance with the DFU andother wound assement, prediction, and therapeutic methods disclosedherein. In some embodiments, images for wound assessment may be capturedwith spectral imaging systems configured to image light within a singlewavelength band. In other embodiments, images may be captured withspectral imaging systems configured to capture two or more wavelengthbads. In one particular example, images may be captured with amonochrome, RGB, and/or infrared imaging device such as those includedin commercially available mobile devices. Further embodiments relate tospectral imaging using a multi-aperture system with curvedmulti-bandpass filters positioned over each aperture. However, it willbe understood that the wound assessment, prediction, and therapeuticmethods of the present technology are not limited to the specific imageacquisition devices disclosed herein, and may equally be implementedwith any imaging device capable of acquiring image data in one or moreknown wavelength bands.

The present disclosure further relates to techniques for implementingspectral unmixing and image registration to generate a spectral datacubeusing image information received from such imaging systems. Thedisclosed technology addresses a number of challenges that are typicallypresent in spectral imaging, described below, in order to yield imagedata that represents precise information about wavelength bands thatwere reflected from an imaged object. In some embodiments, the systemsand methods described herein acquire images from a wide area of tissue(e.g., 5.9×7.9 inches) in a short amount of time (e.g., within 6 secondsor less) and can do so without requiring the injection of imagingcontrast agents. In some aspects, for example, the multispectral imagesystem described herein is configured to acquire images from a wide areaof tissue, e.g., 5.9×7.9 inches, within 6 seconds or less and, whereinsaid multispectral image system is also configured to provide tissueanalysis information, such as identification of a plurality of burnstates, wound states, ulcer states, healing potential, a clinicalcharacteristic including a cancerous or non-cancerous state of theimaged tissue, wound depth, wound volume, a margin for debridement, orthe presence of a diabetic, non-diabetic, or chronic ulcer in theabsence of imaging contrast agents. Similarly, in some of the methodsdescribed herein, the multispectral image system acquires images from awide area of tissue, e.g., 5.9×7.9 inches, within 6 seconds or less andsaid multispectral image system ouputs tissue analysis information, suchas identification of a plurality of burn states, wound states, healingpotential, a clinical characteristic including a cancerous ornon-cancerous state of the imaged tissue, wound depth, wound volume, amargin for debridement, or the presence of a diabetic, non-diabetic, orchronic ulcer in the absence of imaging contrast agents.

One such challenge in existing solutions is that captured images cansuffer from color distortions or disparity that compromise the qualityof the image data. This can be particularly problematic for applicationsthat depend upon precise detection and analysis of certain wavelengthsof light using optical filters. Specifically, color shading is aposition dependent variation in the wavelength of light across the areaof the image sensor, due to the fact that transmittance of a colorfilter shifts to shorter wavelengths as the angle of light incident onthe filter increases. Typically, this effect is observed ininterference-based filters, which are manufactured through thedeposition of thin layers with varying refractive indices onto atransparent substrate. Accordingly, longer wavelengths (such as redlight) can be blocked more at the edges of the image sensor due tolarger incident light ray angles, resulting in the same incomingwavelength of light being detected as a spatially non-uniform coloracross the image sensor. If left uncorrected, color shading manifests asshift in color near the edges of the captured image.

The technology of the present disclosure provides many more benefitsrelative to other multi-spectral imaging systems on the market becauseit is not restrictive in the configuration of lens and/or image sensorsand their respective fields of view or aperture sizes. It will beunderstood that changes to lenses, image sensors, aperture sizes, orother components of the presently disclosed imaging systems may involveother adjustements to the imaging system as would be known to those ofordinary skill in the art. The technology of the present disclosure alsoprovides improvements over other multi-spectral imaging systems in thatthe components that perform the function of resolving wavelengths orcausing the system as a whole to be able to resolve wavelengths (e.g.,optical filters or the like) can be seperable from the components thattransduce light energy into digital outputs (e.g., image sensors or thelike). This reduces the cost, complexity, and/or development time tore-configure imaging systems for different multi-spectral wavelengths.The technology of the present disclosure may be more robust than othermulti-spectral imaging systems in that it can accomplish the sameimaging characteristics as other multi-spectral imaging systems on themarket in a smaller and lighter form factor. The technology of thepresent disclosure is also beneficial relative to other multi-spectralimaging systems in that it can acquire multi-spectral images in asnapshot, video rate, or high speed video rate. The technology of thepresent disclosure also provides a more robust implementation ofmulti-spectral imaging systems based on multi-aperture technology as theability to multiplex several spectral bands into each aperture reducesthe number of apertures necessary to acquire any particular number ofspectral bands in an imaging data set, thus reducing costs through areduced number of apertures and improved light collection (e.g., aslarger apertures may be used within the fixed size and dimensions ofcommercially available sensor arrays). Finally, the technology of thepresent disclosure can provide all of these benefits without a trade-offwith respect to resolution or image quality.

FIG. 1A illustrates an example of a filter 108 positioned along the pathof light towards an image sensor 110, and also illustrates lightincident on the filter 108 at different ray angles. The rays 102A, 104A,106A are represented as lines which, after passing through the filter108, are refracted onto the sensor 110 by a lens 112, which may also besubstituted with any other image-forming optics, including but notlimited to a mirror and/or an aperture. The light for each ray ispresumed in FIG. 1A to be broadband, for example, having a spectralcomposition extending over a large wavelength range to be selectivelyfiltered by filter 108. The three rays 102A, 104A, 106A each arrive atthe filter 108 at a different angle. For illustrative purposes, lightray 102A is shown as being incident substantially normal to filter 108,light ray 104A has a greater angle of incidence than light ray 102A, andlight ray 106A has a greater angle of incidence than light ray 104A. Theresulting filtered rays 102B, 104B, 106B exhibit a unique spectrum dueto the angular dependence of the transmittance properties of the filter108 as seen by the sensor 110. The effect of this dependence causes ashift in the bandpass of the filter 108 towards shorter wavelengths asthe angle of incidence increases. Additionally, the dependence may causea reduction in the transmission efficiency of the filter 108 and analtering of the spectral shape of the bandpass of the filter 108. Thesecombined effects are referred to as the angular-dependent spectraltransmission. FIG. 1B depicts the spectrum of each light ray in FIG. 1Aas seen by a hypothetical spectrometer at the location of sensor 110 toillustrate the shifting of the spectral bandpass of filter 108 inresponse to increasing angle of incidence. The curves 102C, 104C, and106C demonstrate the shortening of the center wavelength of thebandpass; hence, the shortening of the wavelengths of light passed bythe optical system in the example. Also shown, the shape of the bandpassand the peak transmission are altered due to the angle incidence, aswell. For certain consumer applications, image processing can be appliedto remove the visible effects of this angular-dependent spectraltransmission. However, these post-processing techniques do not allow forrecovery of precise information regarding which wavelength of light wasactually incident upon the filter 108. Accordingly, the resulting imagedata may be unusable for certain high-precision applications.

Another challenge faced by certain existing spectral imaging systems isthe time required for capture of a complete set of spectral image data,as discussed in connection with FIGS. 2A and 2B. Spectral imagingsensors sample the spectral irradiance I(x,y,λ) of a scene and thuscollect a three-dimensional (3D) dataset typically called a datacube.FIG. 2A illustrates an example of a spectral image datacube 120. Asillustrated, the datacube 120 represents three dimensions of image data:two spatial dimensions (x and y) corresponding to the two-dimensional(2D) surface of the image sensor, and a spectral dimension (Δ)corresponding to a particular wavelength band. The dimensions of thedatacube 120 can be given by N_(x)N_(y)N_(λ), where N_(x), N_(y), andN_(λ) are the number of sample elements along the (x, y) spatialdimensions and spectral axes A, respectively. Because datacubes are of ahigher dimensionality than 2D detector arrays (e.g., image sensors) thatare currently available, typical spectral imaging systems either capturetime-sequential 2D slices, or planes, of the datacube 120 (referred toherein as “scanning” imaging systems), or simultaneously measure allelements of the datacube by dividing it into multiple 2D elements thatcan be recombined into datacube 120 in processing (referred to herein as“snapshot” imaging systems).

FIG. 2B illustrates examples of how certain scanning spectral imagingtechnologies generate the datacube 120. Specifically, FIG. 2Billustrates the portions 132, 134, and 136 of the datacube 120 that canbe collected during a single detector integration period. A pointscanning spectrometer, for example, can capture a portion 132 thatextends across all spectral planes λ at a single (x, y) spatialposition. A point scanning spectrometer can be used to build thedatacube 120 by performing a number of integrations corresponding toeach (x, y) position across the spatial dimensions. A filter wheelimaging system, for example, can capture a portion 134 that extendsacross the entirety of both spatial dimensions x and y, but only asingle spectral plane A. A wavelength scanning imaging system, such as afilter wheel imaging system, can be used to build the datacube 120 byperforming a number of integrations corresponding to the number ofspectral planes λ. A line scanning spectrometer, for example, cancapture a portion 136 that extends across all spectral dimensions λ andall of one of the spatial dimension (x or y), but only a single pointalong the other spatial dimension (y or x). A line scanning spectrometercan be used to build the datacube 120 by performing a number ofintegrations corresponding to each position of this other spatialdimension (y or x).

For applications in which the target object and imaging system are bothmotionless (or remain relatively still over the exposure times), suchscanning imaging systems provide the benefit of yielding a highresolution datacube 120. For line scanning and wavelength scanningimaging systems, this can be due to the fact that each spectral orspatial image is captured using the entire area of the image sensor.However, movement of the imaging system and/or object between exposurescan cause artifacts in the resulting image data. For example, the same(x, y) position in the datacube 120 can actually represent a differentphysical location on the imaged object across the spectral dimension k.This can lead to errors in downstream analysis and/or impose anadditional requirement for performing registration (e.g., aligning thespectral dimension λ so that a particular (x, y) position corresponds tothe same physical location on the object).

In comparison, a snapshot imaging system 140 can capture an entiredatacube 120 in a single integration period or exposure, therebyavoiding such motion-induced image quality issues. FIG. 2C depicts anexample image sensor 142 and an optical filter array such as a colorfilter array (CFA) 144 that can be used to create a snapshot imagingsystem. The CFA 144 in this example is a repeating pattern of colorfilter units 146 across the surface of the image sensor 142. This methodof acquiring spectral information can also be referred to as amultispectral filter array (MSFA) or a spectrally resolved detectorarray (SRDA). In the illustrated example, the color filter unit 146includes a 5×5 arrangement of different color filters, which wouldgenerate 25 spectral channels in the resulting image data. By way ofthese different color filters, the CFA can split incoming light into thebands of the filters, and direct the split light to dedicatedphotoreceptors on the image sensor. In this way, for a given color 148,only 1/25^(th) of the photoreceptors actually detect a signal representlight of that wavelength. Thus, although 25 different color channels canbe generated in a single exposure with this snapshot imaging system 140,each color channel represents a smaller quantity of measured data thanthe total output of the sensor 142. In some embodiments, a CFA mayinclude one or more of a filter array (MSFA), a spectrally resolveddetector array (SRDA), and/or may include a conventional Bayer filter,CMYK filter, or any other absorption-based or interference-basedfilters. One type of interference based filter would be an array of thinfilm filters arranged in a grid with each element of the gridcorresponding to one or more sensor elements. Another type ofinterference based filter is a Fabry-Perot filter. Nanoetchedinterference Fabry-Perot filters, which exhibit typical bandpassfull-width-at-half-maxima (FWHM) on the order of 20 to 50 nm, areadvantageous because they can be used in some embodiments due to theslow roll-off of the filters' passband seen in the transition from itscenter wavelength to its blocking band. These filters also exhibit a lowOD in these blocking bands further enabling increased sensitivity tolight outside of their passbands. These combined effects makes thesespecific filters sensitive to spectral regions that would otherwise beblocked by the fast roll-off of a high OD interference filter with asimilar FWHM made with many thin film layers in a coating depositionprocess such as in evaporative deposition or in ion-beam sputtering. Inembodiments with dye-based CMYK or RGB (Bayer) filter configurations,the slow spectral roll-off and the large FWHM of individual filterpassbands are preferred and provide a unique spectral transmissionpercentage to individual wavelengths throughout an observed spectrum.

Accordingly, the datacube 120 that results from a snapshot imagingsystem will have one of two properties that can be problematic forprecision imaging applications. As a first option, the datacube 120 thatresults from a snapshot imaging system can have smaller N_(x) and N_(y)sizes than the (x, y) size of the detector array and, thus be of lowerresolution than the datacube 120, which would be generated by a scanningimaging system having the same image sensor. As a second option, thedatacube 120 that results from a snapshot imaging system can have thesame N_(x) and N_(y) sizes as the (x, y) size of the detector array dueto interpolating values for certain (x, y) positions. However, theinterpolation used to generate such a datacube means that certain valuesin the datacube are not actual measurements of the wavelength of lightincident on the sensor, but rather estimates of what the actualmeasurement may be based on surrounding values.

Another existing option for single-exposure multispectral imaging is themultispectral beamsplitter. In such imaging systems, beamsplitter cubessplit incident light into distinct color bands, with each band observedby independent image sensors. While one can change the beamsplitterdesigns to adjust the measured spectral bands, it is not easy to dividethe incident light into more than four beams without compromising thesystem performance. Thus, four spectral channels appear to be thepractical limit of this approach. A closely related method is to usethin-film filters instead of the bulkier beamsplitter cubes/prisms tosplit the light, however this approach is still limited to about sixspectral channels due to space limitations and cumulative transmissionlosses through successive filters.

The aforementioned problems, among others, are addressed in someembodiments by the disclosed multi-aperture spectral imaging systemwith, multi-bandpass filters, preferably curved multi-bandpass filters,to filter light incoming through each aperture, and the associated imagedata processing techniques. This particular configuration is able toachieve all of the design goals of fast imaging speeds, high resolutionimages, and precise fidelity of detected wavelengths. Accordingly, thedisclosed optical design and associated image data processing techniquescan be used in portable spectral imaging systems and/or to image movingtargets, while still yielding a datacube suitable for high precisionapplications (e.g., clinical tissue analysis, biometric recognition,transient clinical events). These higher precision applications mayinclude the diagnosis of melanoma in the preceeding stages (0 through 3)before metastasis, the classification of a wound or burn severity onskin tissue, or the tissue diagnosis of diabetic foot ulcer severity.Accordingly, the small form factor and the snapshot spectral acquisitionas depicted in some embodiments will enable the use of this invention inclinical environments with transient events, which include the diagnosisof several different retinopathies (e.g. non proliferative diabeticretinopathy, proliferative diabetic retinopathy, and age-related maculardegeneration) and the imaging of moving pediatric patients. Accordingly,it will be appreciated by one of skill in the art that the use of amulti-aperture system with flat or curved multi-bandpass filters, asdisclosed herein, represents a significant technological advance overprior spectral imaging implementations. Specifically, the multi-aperturesystem may enable the collection of 3D spatial images of or relating toobject curvature, depth, volume, and/or area based on the calculateddisparity of the perspective differences between each aperture. However,the multi-aperture strategies presented here are not limited to anyspecific filter and may include flat and/or thin filters, based oneither interference or absorptive filtering. This invention, asdisclosed herein, can be modified to include flat filters in the imagespace of the imaging system in the event of suitable lenses or aperturesthat use a small or acceptable range of incidence angles. Filters mayalso be placed at the aperture stop or at the entrance/exit pupil of theimaging lenses as one skilled in the art of optical engineering may seefit to do so.

Various aspects of the disclosure will now be described with regard tocertain examples and embodiments, which are intended to illustrate butnot limit the disclosure. Although the examples and embodimentsdescribed herein will focus, for the purpose of illustration, onspecific calculations and algorithms, one of skill in the art willappreciate the examples are to illustrate only, and are not intended tobe limiting. For example, although some examples are presented in thecontext of a multispectral imaging, the disclosed multi-aperture imagingsystem and associated filters can be configured to achieve hyperspectralimaging in other implementations. Further, although certain examples arepresented as achieving benefits for handheld and/or moving targetapplications, it will be appreciated that the disclosed imaging systemdesign and associated processing techniques can yield a high precisiondatacube suitable for fixed imaging systems and/or for analysis ofrelatively motionless targets.

Overview of Electromagnetic Ranges and Image Sensors

Certain colors or portions of the electromagnetic spectrum are referredto herein, and will now be discussed with respect to their wavelength asdefined by the ISO 21348 definitions of irradiance spectral categories.As described further below, in certain imaging applications thewavelength ranges for specific colors can be grouped together to passthrough a certain filter.

Electromagnetic radiation ranging from wavelengths of or approximately760 nm to wavelengths of or approximately 380 nm are typicallyconsidered the “visible” spectrum, that is, the portion of the spectrumrecognizable by the color receptors of the human eye. Within the visiblespectrum, red light typically is considered to have a wavelength of orapproximately 700 nanometers (nm), or to be in the range of orapproximately 760 nm to 610 nm or approximately 610 nm. Orange lighttypically is considered to have a wavelength of or approximately 600 nm,or to be in the range of or approximately 610 nm to approximately 591 nmor 591 nm. Yellow light typically is considered to have a wavelength ofor approximately 580 nm, or to be in the range of or approximately 591nm to approximately 570 nm or 570 nm. Green light typically isconsidered to have a wavelength of or approximately 550 nm, or to be inthe range of or approximately 570 nm to approximately 500 nm or 500 nm.Blue light typically is considered to have a wavelength of orapproximately 475 nm, or to be in the range of or approximately 500 nmto approximately 450 nm or 450 nm. Violet (purple) light typically isconsidered to have a wavelength of or approximately 400 nm, or to be inthe range of or approximately 450 nm to approximately 360 nm or 360 nm.

Turning to ranges outside of the visible spectrum, infrared (IR) refersto electromagnetic radiation with longer wavelengths than those ofvisible light, and is generally invisible to the human eye. IRwavelengths extend from the nominal red edge of the visible spectrum atapproximately 760 nm or 760 nm to approximately 1 millimeter (mm) or 1mm. Within this range, near infrared (NIR) refers to the portion of thespectrum that is adjacent to the red range, ranging from wavelengthsbetween approximately 760 nm or 760 nm to approximately 1400 nm or 1400nm.

Ultraviolet (UV) radiation refers to some electromagnetic radiation withshorter wavelengths than those of visible light, and is generallyinvisible to the human eye. UV wavelengths extend from the nominalviolet edge of the visible spectrum at approximately 40 nm or 40 nm toapproximately 400 nm. Within this range, near ultraviolet (NUV) refersto the portion of the spectrum that is adjacent to the violet range,ranging from wavelengths between approximately 400 nm or 400 nm toapproximately 300 nm or 300 nm, middle ultraviolet (MUV) ranges fromwavelengths between approximately 300 nm or 300 nm to approximately 200nm or 200 nm, and far ultraviolet (FUV) ranges from wavelengths betweenapproximately 200 nm or 200 nm to approximately 122 nm or 122 nm.

The image sensors described herein can be configured to detectelectromagnetic radiation in any of the above-described ranges,depending upon the particular wavelength ranges that are suitable for aparticular application. The spectral sensitivity of a typicalsilicon-based charge-coupled device (CCD) or complementarymetal-oxide-semiconductor (CMOS) sensor extends across the visiblespectrum, and also extends considerably into the near-infrared (IR)spectrum and sometimes into the UV spectrum. Some implementations canalternatively or additionally use back-illuminated or front-illuminatedCCD or CMOS arrays. For applications requiring high SNR andscientific-grade measurements, some implementations can alternatively oradditionally use either scientific complementarymetal-oxide-semiconductor (sCMOS) cameras or electron multiplying CCDcameras (EMCCD). Other implementations can alternatively or additionallyuse sensors known to operate in specific color ranges (e.g., short-waveinfrared (SWIR), mid-wave infrared (MWIR), or long-wave infrared (LWIR))and corresponding optical filter arryas, based on the intendedapplications. These may alternatively or additionally include camerasbased around detector materials including indium gallium arsenide(InGaAs) or indium antimonide (InSb) or based around microbolometerarrays.

The image sensors used in the disclosed multispectral imaging techniquesmay be used in conjunction with an optical filter array such as a colorfilter array (CFA). Some CFAs can split incoming light in the visiblerange into red (R), green (G), and blue (B) categories to direct thesplit visible light to dedicated red, green, or blue photodiodereceptors on the image sensor. A common example for a CFA is the Bayerpattern, which is a specific pattern for arranging RGB color filters ona rectangular grid of photosensors. The Bayer pattern is 50% green, 25%red and 25% blue with rows of repeating red and green color filtersalternating with rows of repeating blue and green color filters. SomeCFAs (e.g., for RGB-NIR sensors) can also separate out the NIR light anddirect the split NIR light to dedicated photodiode receptors on theimage sensor.

As such, the wavelength ranges of the filter components of the CFA candetermine the wavelength ranges represented by each image channel in acaptured image. Accordingly, a red channel of an image may correspond tothe red wavelength regions of the color filter and can include someyellow and orange light, ranging from approximately 570 nm or 570 nm toapproximately 760 nm or 760 nm in various embodiments. A green channelof an image may correspond to a green wavelength region of a colorfilter and can include some yellow light, ranging from approximately 570nm or 570 nm to approximately 480 nm or 480 nm in various embodiments. Ablue channel of an image may correspond to a blue wavelength region of acolor filter and can include some violet light, ranging fromapproximately 490 nm or 490 nm to approximately 400 nm or 400 nm invarious embodiments. As a person of ordinary skill in the art willappreciate, exact beginning and ending wavelengths (or portions of theelectromagnetic spectrum) that define colors of a CFA (for example, red,green, and blue) can vary depending upon the CFA implementation.

Further, typical visible light CFAs are transparent to light outside thevisible spectrum. Therefore, in many image sensors the IR sensitivity islimited by a thin-film reflective IR filter at the face of the sensorthat blocks the infrared wavelength while passing visible light.However, this may be omitted in some of the disclosed imaging systems toallow of passage of IR light. Thus, the red, green, and/or blue channelsmay also be used to collect IR wavelength bands. In some implementationsthe blue channel may also be used to collect certain NUV wavelengthbands. The distinct spectral responses of the red, green, and bluechannels with regard to their unique transmission efficiencies at eachwavelength in a spectral image stack may provide a uniquely weightedresponse of spectral bands to be unmixed using the known transmissionprofiles. For example, this may include the known transmission responsein IR and UV wavelength regions for the red, blue, and green channels,enabling their use in the collection of bands from these regions.

As described in further detail below, additional color filters can beplaced before the CFA along the path of light towards the image sensorin order to selectively refine the specific bands of light that becomeincident on the image sensor. Some of the disclosed filters can beeither a combination of dichroic (thin-film) and/or absorptive filtersor a single dichroic and/or absorptive filter. Some of the disclosedcolor filters can be bandpass filters that pass frequencies within acertain range (in a passband) and reject (attenuates) frequenciesoutside that range (in a blocking range). Some of the disclosed colorfilters can be multi-bandpass filters that pass multiple discontinuousranges of wavelengths. These “wavebands” can have smaller passbandranges, larger blocking range attenuation, and sharper spectralroll-off, which is defined as the steepness of the spectral response asthe filter transitions from the passband to the blocking range, than thelarger color range of the CFA filter. For example, these disclosed colorfilters can cover a passband of approximately 20 nm or 20 nm orapproximately 40 nm or 40 nm. The particular configuration of such colorfilters can determine the actual wavelength bands that are incident uponthe sensor, which can increase the precision of the disclosed imagingtechniques. The color filters described herein can be configured toselectively block or pass specific bands of electromagnetic radiation inany of the above-described ranges, depending upon the particularwavelength bands that are suitable for a particular application.

As described herein, a “pixel” can be used to describe the outputgenerated by an element of the 2D detector array. In comparison, aphotodiode, a single photosensitive element in this array, behaves as atransducer capable of converting photons into electrons via thephotoelectric effect, which is then in turn converted into a usablesignal used to determine the pixel value. A single element of thedatacube can be referred to as a “voxel” (e.g., a volume element). A“spectral vector” refers to a vector describing the spectral data at aparticular (x, y) position in a datacube (e.g., the spectrum of lightreceived from a particular point in the object space). A singlehorizontal plane of the datacube (e.g., an image representing a singlespectral dimension), is referred to herein as a an “image channel”.Certain embodiments described herein may capture spectral videoinformation, and the resulting data dimensions can assume the“hypercube” form N_(x)N_(y)N_(λ)N_(t), where N_(t) is the number offrames captured during a video sequence.

Overview of Example Multi-Aperture Imaging Systems with CurvedMulti-Bandpass Filters

FIG. 3A depicts a schematic view of an example multi-aperture imagingsystem 200 with curved multi-bandpass filters, according to the presentdisclosure. The illustrated view includes a first image sensor region225A (photodiodes PD1-PD3) and a second image sensor region 225B(photodiodes PD4-PD6). The photodiodes PD1-PD6 can be, for example,photodiodes formed in a semiconductor substrate, for example in a CMOSimage sensor. Generally, each of the photodiodes PD1-PD6 can be a singleunit of any material, semiconductor, sensor element or other device thatconverts incident light into current. It will be appreciated that asmall portion of the overall system is illustrated for the purpose ofexplaining its structure and operation, and that in implementation imagesensor regions can have hundreds or thousands of photodiodes (andcorresponding color filters). The image sensor regions 225A and 225B maybe implemented as separate sensors, or as separate regions of the sameimage sensor, depending upon the implementation. Although FIG. 3Adepicts two apertures and corresponding light paths and sensor regions,it will be appreciated that the optical design principles illustrated byFIG. 3A can be extended to three or more apertures and correspondinglight paths and sensor regions, depending upon the implementation.

The multi-aperture imaging system 200 includes a first opening 210A thatprovides a first light path towards the first sensor region 225A, and asecond opening 210B that provides a first light path towards the secondsensor region 225B. These apertures may be adjustable to increase ordecrease the brightness of the light that falls on the image, or so thatthe duration of particular image exposures can be changed and thebrightness of the light that falls on the image sensor regions does notchange. These apertures may also be located at any position along theoptical axes of this multi-aperture system as deemed reasonable by oneskilled in the art of optical design. The optical axis of the opticalcomponents positioned along the first light path is illustrated bydashed line 230A and the optical axis of the optical componentspositioned along the second light path is illustrated by dashed line230B, and it will be appreciated that these dashed lines do notrepresent a physical structure of the multi-aperture imaging system 200.The optical axes 230A, 230B are separated by a distance D, which canresult in disparity between the images captured by the first and secondsensor regions 225A, 225B. Disparity refers to the distance between twocorresponding points in the left and right (or upper and lower) imagesof a stereoscopic pair, such that the same physical point in the objectspace can appear in different locations in each image. Processingtechniques to compensate for and leverage this disparity are describedin further detail below.

Each optical axis 230A, 230B passes through a center C of thecorresponding aperture, and the optical components can also be centeredalong these optical axes (e.g., the point of rotational symmetry of anoptical component can be positioned along the optical axis). Forexample, the first curved multi-bandpass filter 205A and first imaginglens 215A can be centered along the first optical axis 230A, and thesecond curved multi-bandpass filter 205B and second imaging lens 215Bcan be centered along the second optical axis 230B.

As used herein with respect to positioning of optical elements, “over”and “above” refer to the position of a structure (for example, a colorfilter or lens) such that light entering the imaging system 200 from theobject space propagates through the structure before it reaches (or isincident upon) another structure. To illustrate, along the first lightpath, the curved multi-bandpass filter 205A is positioned above theaperture 210A, the aperture 210A is positioned above imaging lens 215A,the imaging lens 215A is positioned above the CFA 220A, and the CFA 220Ais positioned above the first image sensor region 225A. Accordingly,light from the object space (e.g., the physical space being imaged)first passes through the curved multi-bandpass filter 205A, then theaperture 210A, then the imaging lens 215A, then the CFA 220A, andfinally is incident on the first image sensor region 225A. The secondlight path (e.g., curved multi-bandpass filter 205B, aperture 210B,imaging lens 215B, CFA 220B, second image sensor region 225B) follows asimilar arrangement. In other implementations, the aperture 210A, 210Band/or imaging lenses 215A, 215B can be positioned above the curvedmulti-bandpass filter 205A, 205B. Additionally, other implementationsmay not use a physical aperture and may rely on the clear aperture ofthe optics to control the brightness of light that is imaged onto thesensor region 225A, 225B. Accordingly, the lens 215A, 215B may be placedabove the aperture 210A, 210B and curved multi-bandpass filter 205A,205B. In this implementation, the aperture 210A, 210B and lens 215A,215B may be also be placed over or under each other as deemed necessaryby one skilled in the art of optical design.

The first CFA 220A positioned over the first sensor region 225A and thesecond CFA 220B positioned over the second sensor region 225B can act aswavelength-selective pass filters and split incoming light in thevisible range into red, green, and blue ranges (as indicated by the R,G, and B notation). The light is “split” by allowing only certainselected wavelengths to pass through each of the color filters in thefirst and second CFAs 220A, 220B. The split light is received bydedicated red, green, or blue diodes on the image sensor. Although red,blue, and green color filters are commonly used, in other embodimentsthe color filters can vary according to the color channel requirementsof the captured image data, for example including ultraviolet, infrared,or near-infrared pass filters, as with an RGB-IR CFA.

As illustrated, each filter of the CFA is positioned over a singlephotodiode PD1-PD6. FIG. 3A also illustrates example microlenses(denoted by ML) that can be formed on or otherwise positioned over eachcolor filter, in order to focus incoming light onto active detectorregions. Other implementations may have multiple photodiodes under asingle filter (e.g., clusters of 2, 4, or more adjacent photodiodes). Inthe illustrated example, photodiode PD1 and photodiode PD4 are under redcolor filters and thus would output red channel pixel information;photodiode PD2 and photodiode PD5 are under green color filters and,thus would output green channel pixel information; and photodiode PD3and photodiode PD6 are under blue color filters and thus would outputblue channel pixel information. Further, as described in more detailbelow, the specific color channels output by given photodiodes can befurther limited to narrower wavebands based on activated illuminantsand/or the specific wavebands passed by the multi-bandpass filters 205A,205B, such that a given photodiode can output different image channelinformation during different exposures.

The imaging lenses 215A, 215B can be shaped to focus an image of theobject scene onto the sensor regions 225A, 225B. Each imaging lens 215A,215B may be composed of as many optical elements and surfaces needed forimage formation and are not limited to single convex lenses as presentedin FIG. 3A, enabling the use of a wide variety of imaging lenses or lensassemblies that would be available commercially or by custom design.Each element or lens assembly may be formed or bonded together in astack or held in series using an optomechanical barrel with a retainingring or bezel. In some embodiments, elements or lens assemblies mayinclude one or more bonded lens groups, such as two or more opticalcomponents cemented or otherwise bonded together. In variousembodiments, any of the multi-bandpass filters described herein may bepositioned in front of a lens assembly of the multispectral imagesystem, in front of a singlet of the multispectral image system, behinda lens assembly of the multispectral image system, behind a singlet ofthe multispectral image system, inside a lens assembly of themultispectral image system, inside a bonded lens group of themultispectral image system, directly onto a surface of a singlet of themultispectral image system, or directly onto a surface of an element ofa lens assembly of the multispectral image system. Further, the aperture210A and 210B may be removed, and the lenses 215A, 215B may be of thevariety typically used in photography with eitherdigital-single-lens-reflex (DSLR) or mirrorless cameras. Addtionally,these lenses may be of the variety used in machine vision using C-mountor S-mount threading for mounting. Focus adjustment can be provided bymovement of the imaging lenses 215A, 215B relative to the sensor regions225A, 225B or movement of the sensor regions 225A, 225B relative to theimaging lenses 215A, 215B, for example based on manual focusing,contrast-based autofocus, or other suitable autofocus techniques.

The multi-bandpass filters 205A, 205B can be each configured toselectively pass multiple narrow wavebands of light, for examplewavebands of 10-50 nm in some embodiments (or wider or narrowerwavebands in other embodiments). As illustrated in FIG. 3A, bothmulti-bandpass filters 205A, 205B can pass waveband λ_(c) (the “commonwaveband”). In implementations with three or more light paths, eachmulti-bandpass filter can pass this common waveband. In this manner,each sensor region captures image information at the same waveband (the“common channel”). This image information in this common channel can beused to register the sets of images captured by each sensor region, asdescribed in further detail below. Some implementations may have onecommon waveband and corresponding common channel, or may have multiplecommon wavebands and corresponding common channels.

In addition to the common waveband λ_(c), each multi-bandpass filters205A, 205B can be each configured to selectively pass one or more uniquewavebands. In this manner, the imaging system 200 is able to increasethe number of distinct spectral channels captured collectively by thesensor regions 205A, 205B beyond what can be captured by a single sensorregion. This is illustrated in FIG. 3A by multi-bandpass filters 205Apassing unique waveband λ_(u1), and multi-bandpass filters 205B passingunique waveband λ_(u2), where λ_(u1) and λ_(u2) represent differentwavebands from one another. Although depicted as passing two wavebands,the disclosed multi-bandpass can each pass a set of two or morewavebands. For example, some implementations can pass four wavebandseach, as described with respect to FIGS. 11A and 11B. In variousembodiments, a larger number of wavebands may be passed. For example,some four-camera implementations may include multi-bandpass filtersconfigured to pass 8 wavebands. In some embodiments, the number ofwavebands may be, for example, 4, 5, 6, 7, 8, 9, 10, 12, 15, 16, or morewavebands.

The multi-bandpass filters 205A, 205B have a curvature selected toreduce the angular-dependent spectral transmission across the respectivesensor regions 225A, 225B. As a result, when receiving narrowbandillumination from the object space, each photodiode across the area ofthe sensor regions 225A, 225B that is sensitive to that wavelength(e.g., the overlying color filter passes that wavelength) should receivesubstantially the same wavelength of light, rather than photodiodes nearthe edge of the sensor experiencing the wavelength shift described abovewith respect to FIG. 1A. This can generate more precise spectral imagedata than using flat filters.

FIG. 3B depicts an example optical design for optical components of onelight path of the multi-aperture imaging system of FIG. 3A.Specifically, FIG. 3B depicts a custom achromatic doublet 240 that canbe used to provide the multi-bandpass filters 205A, 205B. The customachromatic doublet 240 passes light through a housing 250 to an imagesensor 225. The housing 250 can include openings 210A, 210B and imaginglens 215A, 215B described above.

The achromatic doublet 240 is configured to correct for opticalabberations as introduced by the incorporation of surfaces required forthe multi-bandpass filter coatings 205A, 205B. The illustratedachromatic doublet 240 includes two individual lenses, which can be madefrom glasses or other optical materials having different amounts ofdispersion and different refractive indicies. Other implementations mayuse three or more lenses. These achromatic doublet lenses can bedesigned to incorporate the multi-bandpass filter coatings 205A, 205B onthe curved front surface 242 while eliminating optical aberrationsintroduced that would otherwise be present through the incorporation ofa curved singlet optical surface with the deposited filter coatings205A, 205B while still limiting optical or focusing power provided bythe achromatic doublet 240 due to the combinatorial effect of the curvedfront surface 242 and the curved back surface of 244 while still keepingthe primary elements for focusing light restricted to the lenses housedin housing 250. Thus, the achromatic doublet 240 can contribute to thehigh precision of image data captured by the system 200. Theseindividual lenses can be mounted next to each other, for example beingbonded or cemented together, and shaped such that the aberration of oneof the lenses is counterbalanced by that of the other. The achromaticdoublet 240 curved front surface 242 or the curved back surface 244 canbe coated with the multi-bandpass filter coating 205A, 205B. Otherdoublet designs may be implemented with the systems described herein.

Further variations of the optical designs described herein may beimplemented. For example, in some embodiments a light path may include asinglet or other optical singlet such as of the positive or negativemeniscus variety as depicted in FIG. 3A instead of the doublet 240depicted in FIG. 3B. FIG. 3C illustrates an example implementation inwhich a flat filter 252 is included between the lens housing 250 and thesensor 225. The achromatic doublet 240 in FIG. 3C provides opticalaberration correction as introduced by the inclusion of the flat filter252 containing a multi-bandpass transmission profile while notsignificantly contributing to the optical power as provided by thelenses contained in housing 250. FIG. 3D illustrates another example ofan implementation in which the multi-bandpass coating is implemented bymeans of a multi-bandpass coating 254 applied to the front surface ofthe lens assembly contained within the housing 250. As such, thismulti-bandpass coating 254 may be applied to any curved surface of anyoptical element residing within housing 250.

FIGS. 4A-4E depict an embodiment of a multispectral, multi-apertureimaging system 300, with an optical design as described with respect toFIGS. 3A and 3B. Specifically, FIG. 4A depicts a perspective view of theimaging system 300 with the housing 305 illustrated with translucency toreveal interior components. The housing 305 may be larger or smallerrelative to the illustrated housing 305, for example, based on a desiredamount of embedded computing resources. FIG. 4B depicts a front view ofthe imaging system 300. FIG. 4C depicts a cutaway side view of theimaging system 300, cut along line C-C illustrated in FIG. 4B. FIG. 4Ddepicts a bottom view of the imaging system 300 depicting the processingboard 335. FIGS. 4A-4D are described together below.

The housing 305 of the imaging system 300 may be encased in anotherhousing. For example, handheld implementations may enclose the systemwithin a housing optionally with one or more handles shaped tofacilitate stable holding of the imaging system 300. Example handheldimplementations are depicted in greater detail in FIGS. 18A-18C and inFIGS. 19A-19B. The upper surface of the housing 305 includes fouropenings 320A-320D. A different multi-bandpass filter 325A-325D ispositioned over each opening 320A-320D and held in place by a filter cap330A-330B. The multi-bandpass filters 325A-325D may or may not becurved, and each pass a common waveband and at least one uniquewaveband, as described herein, in order to achieve high precisionmulti-spectral imaging across a greater number of spectral channels thanwould otherwise be captured by the image sensor due to its overlyingcolor filter array. The image sensor, imaging lenses, and color filtersdescribed above are positioned within the camera housings 345A-345D. Insome embodiments, a single camera housing may enclose the image sensors,imaging lenses, and color filters described above, for example, as shownin FIGS. 20A-20B. In the depicted implementation separate sensors arethus used (e.g., one sensor within each camera housing 345A-345D), butit will be appreciated that a single image sensor spanning across all ofthe regions exposed through the openings 320A-320D could be used inother implementations. The camera housings 345A-345D are secured to thesystem housing 305 using supports 340 in this embodiment, and can besecured using other supports in various implementations.

The upper surface of the housing 305 supports an optional illuminationboard 310 covered by an optical diffusing element 315. The illuminationboard 310 is described in further detail with respect to FIG. 4E, below.The diffusing element 315 can be composed of glass, plastic, or otheroptical material for diffusing light emitted from the illumination board310 such that the object space receives substantially spatially-evenillumination. Even illumination of the target object can be beneficialin certain imaging applications, for example clinical analysis of imagedtissue, because it provides, within each wavelength, a substantiallyeven amount of illumination across the object surface. In someembodiments, the imaging systems disclosed herein may utilize ambientlight instead of or in addition to light from the optional illuminationboard.

Due to heat generated by the illumination board 310 in use, the imagingsystem 300 includes a heat sink 350 including a number of heatdissipating fins 355. The heat dissipating fins 355 can extend into thespace between the camera housings 345A-345D, and the upper portion ofthe heat sink 350 can draw heat from the illumination board 310 to thefins 355. The heat sink 350 can be made from suitable thermallyconductive materials. The heat sink 350 may further help to dissipateheat from other components such that some implementations of imagingsystems may be fanless.

A number of supports 365 in the housing 305 secure a processing board335 in communication with the cameras 345A-345D. The processing board335 can control operation of the imaging system 300. Although notillustrated, the imaging system 300 can also be configured with one ormore memories, for example storing data generated by use of the imagingsystem and/or modules of computer-executable instructions for systemcontrol. The processing board 335 can be configured in a variety ofways, depending upon system design goals. For example, the processingboard can be configured (e.g., by a module of computer-executableinstructions) to control activation of particular LEDs of theillumination board 310. Some implementations can use a highly stablesynchronous step-down LED driver, which can enable software control ofanalog LED current and also detect LED failure. Some implementations canadditionally provide image data analysis functionality to the processingboard (e.g., by modules of computer-executable instructions) 335 or to aseparate processing board. Although not illustrated, the imaging system300 can include data interconnects between the sensors and theprocessing board 335 such that the processing board 335 can receive andprocess data from the sensors, and between the illumination board 310and the processing board 335 such that the processing board can driveactivation of particular LEDs of the illumination board 310.

FIG. 4E depicts an example illumination board 310 that may be includedin the imaging system 300, in isolation from the other components. Theillumination board 310 includes four arms extending from a centralregion, with LEDs positioned along each arm in three columns. The spacesbetween LEDs in adjacent columns are laterally offset from one anotherto create separation between adjacent LEDs. Each column of LEDs includesa number of rows having different colors of LEDs. Four green LEDs 371are positioned in the center region, with one green LED in each cornerof the center region. Starting from the innermost row (e.g., closest tothe center), each column includes a row of two deep red LEDs 372 (for atotal of eight deep red LEDs). Continuing radially outward, each arm hasa row of one amber LED 374 in the central column, a row of two shortblue LEDs 376 in the outermost columns (for a total of eight short blueLEDs), another row of one amber LED 374 in the central column (for atotal of eight amber LEDs), a row having one non-PPG NIR LED 373 and onered LED 375 in the outermost columns (for a total of four of each), andone PPG NIR LED 377 in the central column (for a total of four PPG NIRLEDs). A “PPG” LED refers to an LED activated during a number ofsequential exposure for capturing photoplethysmographic (PPG)information representing pulsatile blood flow in living tissue. It willbe understood that a variety of other colors and/or arrangements thereofmay be used in illumination boards of other embodiments.

FIG. 5 depicts another embodiment of a multispectral multi-apertureimaging system, with an optical design as described with respect toFIGS. 3A and 3B. Similar to the design of the imaging system 300, theimaging system 400 includes four light paths, here shown as openings420A-420D having multi-bandpass filter lens groups 425A-425D, which aresecured to housing 405 by retaining rings 430A-430D. The imaging system400 also includes an illumination board 410 secured to the front face ofthe housing 405 between the retaining rings 430A-430D, and a diffuser415 positioned over the illumination board 410 to assist with emittingspatially even light onto the target object.

The illumination board 410 of the system 400 includes four branches ofLEDs in a cross shape, with each branch including two columns ofclosely-spaced LEDs. Thus, the illumination board 410 is more compactthan the illumination board 310 described above, and may be suitable foruse with imaging systems having smaller form factor requirements. Inthis example configuration, each branch includes an outermost row havingone green LED and one blue LED, and moving inwards includes two rows ofyellow LEDs, a row of orange LEDs, a row having one red LED and one deepred LED, and a row having one amber LED and one NIR LED. Accordingly, inthis implementation the LEDs are arranged such that LEDs that emit lightof longer wavelengths are in the center of the illumination board 410,while LEDs that emit light of shorter wavelengths are at the edges ofthe illumination board 410.

FIGS. 6A-6C depict another embodiment of a multispectral multi-apertureimaging system 500, with an optical design as described with respect toFIGS. 3A and 3B. Specifically, FIG. 6A depicts a perspective view of theimaging system 500, FIG. 6B depicts a front view of the imaging system500, and FIG. 6C depicts a cutaway side view of the imaging system 500,cut along line C-C illustrated in FIG. 6B. The imaging system 500includes similar components to those described above with respect toimaging system 300 (e.g., a housing 505, illumination board 510,diffusing plate 515, multi-bandpass filters 525A-525D secured overopenings via retaining rings 530A-530D), but depicts a shorter formfactor (e.g., in an embodiment with fewer and/or smaller embeddedcomputing components). The system 500 also includes a directcamera-to-frame mount 540 for added rigidity and robustness of cameraalignment.

FIGS. 7A-7B depict another embodiment of a multispectral multi-apertureimaging system 600. FIGS. 7A-7B illustrate another possible arrangementof light sources 610A-610C around a multi-aperture imaging system 600.As depicted, four lens assemblies with multi-bandpass filters 625A-625Dwith an optical design as described with respect to FIGS. 3A-3D can bedisposed in a rectangular or square configuration to provide light tofour cameras 630A-630D (including image sensors). Three rectangularlight emitting elements 610A-610C can be disposed parallel to oneanother outside of and between the lens assemblies with multi-bandpassfilters 625A-625D. These can be broad-spectrum light emitting panels orarrangements of LEDs that emit discrete wavebands of light.

FIGS. 8A-8B depict another embodiment of a multispectral multi-apertureimaging system 700. FIGS. 8A-8B illustrate another possible arrangementof light sources 710A-710D around a multi-aperture imaging system 700.As depicted, four lens assemblies with multi-bandpass filters 725A-725D,employing an optical design as described with respect to FIGS. 3A-3D,can be disposed in a rectangular or square configuration to providelight to four cameras 730A-730D (including image sensors). The fourcameras 730A-730D are illustrated in a closer example configurationwhich may minimize perspective differences between the lenses. Fourrectangular light emitting elements 710A-710D can be positioned in asquare surrounding the lens assemblies with multi-bandpass filters725A-725D. These can be broad-spectrum light emitting panels orarrangements of LEDs that emit discrete wavebands of light.

FIGS. 9A-9C depict another embodiment of a multispectral multi-apertureimaging system 800. The imaging system 800 includes a frame 805 coupledto a lens cluster frame front 830 that includes openings 820 and supportstructures for micro-video lenses 825, which can be provided withmulti-bandpass filters using an optical design as described with respectto FIGS. 3A-3D. The micro-video lenses 825 provide light to four cameras845 (including imaging lenses and image sensor regions) mounted on alens cluster frame back 840. Four linear arrangements of LEDs 811 aredisposed along the four sides of the lens cluster frame front 830, eachprovided with its own diffusing element 815. FIGS. 9B and 9C depictexample dimensions in inches to show one possible size of themulti-aperture imaging system 800.

FIG. 10A depicts another embodiment of a multispectral multi-apertureimaging system 900, with an optical design as described with respect toFIGS. 3A-3D. The imaging system 900 can be implemented as a set ofmulti-bandpass filters 905 that are attachable over a multi-aperturecamera 915 of a mobile device 910. For example, certain mobile devices910 such as smartphones can be equipped with stereoscopic imagingsystems having two openings leading to two image sensor regions. Thedisclosed multi-aperture spectral imaging techniques can be implementedin such devices by providing them with a suitable set of multi-bandpassfilters 905 to pass multiple narrower wavebands of light to the sensorregions. Optionally, the set of multi-bandpass filters 905 can beequipped with an illuminant (such as an LED array and diffuser) thatprovides light at these wavebands to the object space.

The system 900 can also include a mobile application that configures themobile device to perform the processing that generates the multispectraldatacube, as well as processing the multispectral datacube (e.g., forclinical tissue classification, biometric recognition, materialsanalysis, or other applications). Alternatively, the mobile applicationmay configure the device 910 to send the multispectral datacube over anetwork to a remote processing system, and then receive and display aresult of the analysis. An example user interface 910 for such anapplication is shown in FIG. 10B.

FIGS. 11A-11B depict an example set of wavebands that can be passed bythe filters of four-filter implementations of the multispectralmulti-aperture imaging systems of FIGS. 3A-10B, for example to an imagesensor having the Bayer CFA (or another RGB or RGB-IR CFA). The spectraltransmission response of wavebands as passed by the multi-bandpassfilters are shown by the solid lines in the graphs 1000 of FIG. 11A andare denotied by T_(n) ^(λ), where n represents the camera number,ranging from 1 through 4. The dashed lines represent the combinedspectral response of T_(n) ^(λ) with either the spectral transmission ofa green pixel, Q_(G) ^(λ), a red pixel, Q_(R) ^(λ), or a blue pixel,Q_(B) ^(λ), that would be present in a typical Bayer CFA. Thesetransmission curves also include the effects of quantum efficiency dueto the sensor used in this example. As illustrated, this set of fourcameras collectively captures eight unique channels or wavebands. Eachfilter passes two common wavebands (the two left-most peaks) to therespective cameras, as well as two additional wavebands. In thisimplementation, the first and third cameras receive light in a firstshared NIR waveband (the right-most peak), and the second and fourthcameras receive light in a second shared NIR waveband (the peaksecond-most to the right). Each of the cameras also receives one uniquewaveband ranging from approximately 550 nm or 550 nm to approximately800 nm or 800 nm. Thus, the camera can capture eight unique spectralchannels using a compact configuration. A graph 1010 in FIG. 11B depictsthe spectral irradiance of an LED board as described in FIG. 4E that maybe used as illumination for the 4 cameras d shown in FIG. 11A.

In this implementation, the eight wavebands have been selected based onproducing spectral channels suitable for clinical tissue classification,and may also be optimized with respect to signal-to-noise ratio (SNR)and frame rate while limiting the number of LEDs (which introduce heatinto the imaging system). The eight wavebands include a common wavebandof blue light (the leftmost peak in the graphs 1000) that is passed byall four filters, because tissue (e.g., animal tissue including humantissue) exhibits higher contrast at blue wavelengths than at green orred wavelengths. Specifically, human tissue exhibits its highestcontrast when imaged at a waveband centered on around 420 nm, as shownin the graphs 1000. Because the channel corresponding to the commonwaveband is used for disparity correction, this higher contrast canproduce more accurate correction. For example in disparity correctionthe image processor can employ local or global methods to find a set ofdisparities so that a figure of merit corresponding to similaritybetween local image patches or images is maximized. Alternatively, theimage processor can employ similar methods that minimize a figure ofmerit corresponding to dissimilarity. These figures of merit can bebased on entropy, correlation, absolute differences, or on deep learningmethods. Global methods of disparity calculation can operateiteratively, terminating when the figure of merit is stable. Localmethods can be used to calculate disparity point by point, using a fixedpatch in one image as an input into the figure of merit and a number ofdifferent patches, each determined by a different value of disparityunder test, from the other image. All such methods can have constraintsimposed on the range of disparities that are considered. Theseconstraints can be based on knowledge of the object depth and distance,for instance. The constraints could also be imposed based on a range ofgradients expected in an object. Constraints on the calculateddisparities can also be imposed by projective geometry, such as theepipolar constraint. Disparity can be calculated at multipleresolutions, with the output of disparities calculated at lowerresolutions acting as initial values or constraints on the disparitiescalculated at the next level of resolution. For instance, a disparitycalculated at a resolution level of 4 pixels in one calculation can beused to set constraints of ±4 pixels in a next calculation of disparityat higher resolution. All algorithms that calculate from disparity willbenefit from higher contrast, particularly if that source of contrast iscorrelated for all viewpoints. Generally speaking, the common wavebandcan be selected based on corresponding to the highest contrast imagingof the material that is expected to be imaged for a particularapplication.

After image capture, color separation between adjacent channels may notbe perfect, and so this implementation also has an additional commonwaveband passed by all filters—depicted in the graphs 1000 as the greenwaveband adjacent to the blue waveband. This is because blue colorfilter pixels are sensitive to retions of the green spectrum due to itsbroad spectral bandpass. This typically manifests as spectral overlap,which may also be characterized as intentional crosstalk, betweenadjacent RGB pixels. This overlap enables the spectral sensitivity ofcolor cameras to be similar to the spectral sensitivity of a humanretina, such that the resultant color space is qualitatively similar tohuman vision. Accordingly, having a common green channel can enableseparation of the portion of the signal generated by blue photodiodesthat truly corresponds to received blue light, by separating out theportion of the signal due to green light. This can be accomplished usingspectral unmixing algorithms that factor in the transmittance (shown inthe legend by T with a solid black line) of the multi-band pass filter,the transmittance of the corresponding CFA color filter (shown in thelegend by Q with dashed red, green, and blue lines). It will beappreciated that some implementations may use red light as a commonwaveband, and in such instances a second common channel may not benecessary.

FIG. 12 illustrates a high-level block diagram of an example compactimaging system 1100 with high resolution spectral imaging capabilities,the system 1100 having a set of components including a processor 1120linked to an multi-aperture spectral camera 1160 and illuminant(s) 1165.A working memory 1105, storage 1110, electronic display 1125, and memory1130 are also in communication with the processor 1120. As describedherein, the system 1100 may capture a greater number of image channelsthan there are different colors of filters in the CFA of the imagesensor by using different multi-bandpass filters placed over differentopenings of the multi-aperture spectral camera 1160.

System 1100 may be a device such as cell phone, digital camera, tabletcomputer, personal digital assistant, or the like. System 1100 may alsobe a more stationary device such as a desktop personal computer, videoconferencing station, or the like that uses an internal or externalcamera for capturing images. System 1100 can also be a combination of animage capture device and a separate processing device receiving imagedata from the image capture device. A plurality of applications may beavailable to the user on system 1100. These applications may includetraditional photographic applications, capture of still images andvideo, dynamic color correction applications, and brightness shadingcorrection applications, among others.

The image capture system 1100 includes the multi-aperture spectralcamera 1160 for capturing images. The multi-aperture spectral camera1160 can be, for example, any of the devices of FIGS. 3A-10B. Themulti-aperture spectral camera 1160 may be coupled to the processor 1120to transmit captured images in different spectral channels and fromdifferent sensor regions to the image processor 1120. The illuminant(s)1165 can also be controlled by the processor to emit light at certainwavelengths during certain exposures, as described in more detail below.The image processor 1120 may be configured to perform various operationson a received captured image in order to output a high quality,disparity corrected multispectral datacube.

Processor 1120 may be a general purpose processing unit or a processorspecially designed for imaging applications. As shown, the processor1120 is connected to a memory 1130 and a working memory 1105. In theillustrated embodiment, the memory 1130 stores a capture control module1135, datacube generation module 1140, datacube analysis module 1145,and operating system 1150. These modules include instructions thatconfigure the processor to perform various image processing and devicemanagement tasks. Working memory 1105 may be used by processor 1120 tostore a working set of processor instructions contained in the modulesof memory 1130. Alternatively, working memory 1105 may also be used byprocessor 1120 to store dynamic data created during the operation ofdevice 1100.

As mentioned above, the processor 1120 is configured by several modulesstored in the memory 1130. The capture control module 1135 includesinstructions that configure the processor 1120 to adjust the focusposition of the multi-aperture spectral camera 1160, in someimplementations. The capture control module 1135 also includesinstructions that configure the processor 1120 to capture images withthe multi-aperture spectral camera 1160, for example multispectralimages captured at different spectral channels as well as PPG imagescaptured at the same spectral channel (e.g., a NIR channel). Non-contactPPG imaging normally uses near-infrared (NIR) wavelengths asillumination to take advantage of the increased photon penetration intothe tissue at this wavelength. Therefore, processor 1120, along withcapture control module 1135, multi-aperture spectral camera 1160, andworking memory 1105 represent one means for capturing a set of spectralimages and/or a sequence of images.

The datacube generation module 1140 includes instructions that configurethe processor 1120 to generate a multispectral datacube based onintensity signals received from the photodiodes of different sensorregions. For example, the datacube generation module 1140 can estimate adisparity between the same regions of an imaged object based on aspectral channel corresponding to the common waveband passed by allmulti-bandpass filters, and can use this disparity to register allspectral images across all captured channels to one another (e.g., suchthat the same point on the object is represented by substantially thesame (x,y) pixel location across all spectral channels). The registeredimages collectively form the multispectral datacube, and the disparityinformation may be used to determine depths of different imaged objects,for example a depth difference between healthy tissue and a deepestlocation within a wound site. In some embodiments, the datacubegeneration module 1140 may also perform spectral unmixing to identifywhich portions of the photodiode intensity signals correspond to whichpassed wavebands, for example based on spectral unmixing algorithms thatfactor in filter transmittances and sensor quantum efficiency.

The datacube analysis module 1145 can implement various techniques toanalyze the multispectral datacube generated by the datacube generationmodule 1140, depending upon the application. For example, someimplementations of the datacube analysis module 1145 can provide themultispectral datacube (and optionally depth information) to a machinelearning model trained to classify each pixel according to a certainstate. These states may be clinical states in the case of tissueimaging, for example burn states (e.g., first degree burn, second degreeburn, third degree burn, or healthy tissue categories), wound states(e.g., hemostasis, inflammation, proliferation, remodeling or healthyskin categories), healing potential (e.g., a score reflecting thelikelihood that the tissue will heal from a wounded state, with orwithout a particular therapy), perfusion states, cancerous states, orother wound-related tissue states. The datacube analysis module 1145 canalso analyze the multispectral datacube for biometric recognition and/ormaterials analysis.

Operating system module 1150 configures the processor 1120 to manage thememory and processing resources of the system 1100. For example,operating system module 1150 may include device drivers to managehardware resources such as the electronic display 1125, storage 1110,multi-aperture spectral camera 1160, or illuminant(s) 1165. Therefore,in some embodiments, instructions contained in the image processingmodules discussed above may not interact with these hardware resourcesdirectly, but instead interact through standard subroutines or APIslocated in operating system component 1150. Instructions withinoperating system 1150 may then interact directly with these hardwarecomponents.

The processor 1120 may be further configured to control the display 1125to display the captured images and/or a result of analyzing themultispectral datacube (e.g., a classified image) to a user. The display1125 may be external to an imaging device including the multi-aperturespectral camera 1160 or may be part of the imaging device. The display1125 may also be configured to provide a view finder for a user prior tocapturing an image. The display 1125 may comprise an LCD or LED screen,and may implement touch sensitive technologies.

Processor 1120 may write data to storage module 1110, for example datarepresenting captured images, multispectral datacubes, and datacubeanalysis results. While storage module 1110 is represented graphicallyas a traditional disk device, those with skill in the art wouldunderstand that the storage module 1110 may be configured as any storagemedia device. For example, the storage module 1110 may include a diskdrive, such as a floppy disk drive, hard disk drive, optical disk driveor magneto-optical disk drive, or a solid state memory such as a FLASHmemory, RAM, ROM, and/or EEPROM. The storage module 1110 can alsoinclude multiple memory units, and any one of the memory units may beconfigured to be within the image capture device 1100, or may beexternal to the image capture system 1100. For example, the storagemodule 1110 may include a ROM memory containing system programinstructions stored within the image capture system 1100. The storagemodule 1110 may also include memory cards or high speed memoriesconfigured to store captured images which may be removable from thecamera.

Although FIG. 12 depicts a system comprising separate components toinclude a processor, imaging sensor, and memory, one skilled in the artwould recognize that these separate components may be combined in avariety of ways to achieve particular design objectives. For example, inan alternative embodiment, the memory components may be combined withprocessor components to save cost and improve performance.

Additionally, although FIG. 12 illustrates two memory components—memorycomponent 1130 comprising several modules and a separate memory 1105comprising a working memory—one with skill in the art would recognizeseveral embodiments utilizing different memory architectures. Forexample, a design may utilize ROM or static RAM memory for the storageof processor instructions implementing the modules contained in memory1130. Alternatively, processor instructions may be read at systemstartup from a disk storage device that is integrated into system 1100or connected via an external device port. The processor instructions maythen be loaded into RAM to facilitate execution by the processor. Forexample, working memory 1105 may be a RAM memory, with instructionsloaded into working memory 1105 before execution by the processor 1120.

Overview of Example Image Processing Techniques

FIG. 13 is a flowchart of an example process 1200 for capturing imagedata using the multispectral multi-aperture imaging systems of FIGS.3A-10B and 12 . FIG. 13 depicts four example exposures that can be usedto generate a multispectral datacube as described herein—a visibleexposure 1205, an additional visible exposure 1210, a non-visibleexposure 1215, and an ambient exposure 1220. It will be appreciated thatthese may be captured in any order, and some exposures may be optionallyremoved from or added to a particular workflow as described below.Further, the process 1200 is described with reference to the wavebandsof FIGS. 11A and 11B, however similar workflows can be implemented usingimage data generated based on other sets of wavebands. Additionally,flat field correction may further be implemented in accordance withvarious known flat field correction techniques, to improve imageacquisition and/or disparity correction in various embodiments.

For the visible exposure 1205, LEDs of first five peaks (the left fivepeaks corresponding to visible light in the graphs 1000 of FIG. 11A) canbe turned on by a control signal to the illumination board. The wave oflight output may need to stabilize, at a time specific to particularLEDs, for example 10 ms. The capture control module 1135 can begin theexposure of the four cameras after this time and can continue thisexposure for a duration of around 30 ms, for example. Thereafter, thecapture control module 1135 can cease the exposure and pull the data offof the sensor regions (e.g., by transferring raw photodiode intensitysignals to the working memory 1105 and/or data store 1110). This datacan include a common spectral channel for use in disparity correction asdescribed herein.

In order to increase the SNR, some implementations can capture theadditional visible exposure 1210 using the same process described forthe visible exposure 1205. Having two identical or near-identicalexposures can increase the SNR to yield more accurate analysis of theimage data. However, this may be omitted in implementations where theSNR of a single image is acceptable. A duplicate exposure with thecommon spectral channel may also enable more accurate disparitycorrection in some implementations.

Some implementations can also capture a non-visible exposure 1215corresponding to NIR or IR light. For example, the capture controlmodule 1135 can activate two different NIR LEDs corresponding to the twoNIR channels shown in FIG. 11A. The wave of light output may need tostabilize, at a time specific to particular LEDs, for example 10 ms. Thecapture control module 1135 can begin the exposure of the four camerasafter this time and continue this exposure for a duration of around 30ms, for example. Thereafter, the capture control module 1135 can ceasethe exposure and pull the data off of the sensor regions (e.g., bytransferring raw photodiode intensity signals to the working memory 1105and/or data store 1110). In this exposure, there may be no commonwaveband passed to all sensor regions, as it can safely be assumed thatthere is no change in the shape or positioning of the object relative tothe exposures 1205, 1210 and, thus previously computed disparity valuescan be used to register the NIR channels.

In some implementations, multiple exposures can be captured sequentiallyto generate PPG data representing the change in shape of a tissue sitedue to pulsatile blood flow. These PPG exposures may be captured at anon-visible wavelength in some implementations. Although the combinationof PPG data with multispectral data may increase the accuracy of certainmedical imaging analyses, the capture of PPG data can also introduceadditional time into the image capture process. This additional time canintroduce errors due to movement of the handheld imager and/or object,in some implementations. Thus, certain implementations may omit captureof PPG data.

Some implementations can additionally capture the ambient exposure 1220.For this exposure, all LEDs can be turned off to capture an image usingambient illumination (e.g., sunlight, light from other illuminantsources). The capture control module 1135 can begin the exposure of thefour cameras after this time and can keep the exposure ongoing for adesired duration of, for example, around 30 ms. Thereafter, the capturecontrol module 1135 can cease the exposure and pull the data off of thesensor regions (e.g., by transferring raw photodiode intensity signalsto the working memory 1105 and/or data store 1110). The intensity valuesof the ambient exposure 1220 can be subtracted from the values of thevisible exposure 1205 (or the visible exposure 1205 corrected for SNR bythe second exposure 1210) and also from the non-visible exposure 1215 inorder to remove the influence of ambient light from the multispectraldatacube. This can increase the accuracy of downstream analysis byisolating the portion of the generated signals that represent lightemitted by the illuminants and reflected from the object/tissue site.Some implementations may omit this step if analytical accuracy issufficient using just the visible 1205, 1210 and non-visible 1215exposures.

It will be appreciated that the particular exposure times listed aboveare examples of one implementation, and that in other implementationsexposure time can vary depending upon the image sensor, illuminantintensity, and imaged object.

FIG. 14 depicts a schematic block diagram of a workflow 1300 forprocessing image data, for example image data captured using the process1200 of FIG. 13 and/or using the multispectral multi-aperture imagingsystems of FIGS. 3A-10B and 12 . The workflow 1300 shows the output oftwo RGB sensor regions 1301A, 1301B, however the workflow 1300 can beextended to greater numbers of sensor regions and sensor regionscorresponding to different CFA color channels.

The RGB sensor outputs from the two sensor regions 1301A, 1301B arestored at the 2D sensor outputs modules 1305A, 1305B, respectively. Thevalues of both sensor regions are sent to the non-linear mapping modules1310A, 1310B, which can perform disparity correction by identifyingdisparity between the captured images using the common channel and thenapplying this determined disparity across all channels to register allspectral images to one another.

The outputs of both non-linear mapping modules 1310A, 1310B are thenprovided to the depth calculation module 1335, which can compute a depthof a particular region of interest in the image data. For example, thedepth may represent the distance between the object and the imagesensor. In some implementations, multiple depth values can be computedand compared to determine the depth of the object relative to somethingother than the image sensor. For example, a greatest depth of a woundbed can be determined, as well as a depth (greatest, lowest, or average)of healthy tissue surrounding the wound bed. By subtracting the depth ofthe healthy tissue from the depth of the wound bed, the deepest depth ofthe wound can be determined. This depth comparison can additionally beperformed at other points in the wound bed (e.g., all or somepredetermined sampling) in order to build a 3D map of the depth of thewound at various points (shown in FIG. 14 as z(x,y) where z would be adepth value). In some embodiments, greater disparity may improve thedepth calculation, although greater disparity may also result in morecomputationally intensive algorithms for such depth calculations.

The outputs of both non-linear mapping modules 1310A, 1310B are alsoprovided to the linear equations module 1320, which can treat the sensedvalues as set of linear equations for spectral unmixing. Oneimplementation can use the Moore-Penrose pseudo-inverse equation as afunction of at least sensor quantum efficiency and filter transmittancevalues to compute actual spectral values (e.g., intensity of light atparticular wavelengths that were incident at each (x,y) image point).This can be used in implementations that require high accuracy, such asclinical diagnostics and other biological applications. Application ofthe spectral unmixing can also provide an estimate of photon flux andSNR.

Based on the disparity-corrected spectral channel images and thespectral unmixing, the workflow 1300 can generate a spectral datacube1325, for example in the illustrated format of F(x,y,λ) where Frepresents the intensity of light at a specific (x,y) image location ata specific wavelength or waveband k.

FIG. 15 graphically depicts disparity and disparity correction forprocessing image data, for example image data captured using the processof FIG. 13 and/or using the multispectral multi-aperture imaging systemsof FIGS. 3A-10B and 12 . The first set of images 1410 show image data ofthe same physical location on an object as captured by four differentsensor regions. As illustrated, this object location is not in the samelocation across the raw images, based on the (x,y) coordinate frames ofthe photodiode grids of the image sensor regions. The second set ofimages 1420 shows that same object location after disparity correction,which is now in the same (x,y) location in the coordinate frame of theregistered images. It will be appreciated that such registration mayinvolve cropping certain data from edge regions of the images that donot entirely overlap with one another.

FIG. 16 graphically depicts a workflow 1500 for performing pixel-wiseclassification on multispectral image data, for example image datacaptured using the process of FIG. 13 , processed according to FIGS. 14and 15 , and/or using the multispectral multi-aperture imaging systemsof FIGS. 3A-10B and 12 .

At block 1510, the multispectral multi-aperture imaging system 1513 cancapture image data representing physical points 1512 on an object 1511.In this example, the object 1511 includes tissue of a patient that has awound. A wound can comprise a burn, a diabetic ulcer (e.g., a diabeticfoot ulcer), a non-diabetic ulcer (e.g., pressure ulcers or slow-healingwounds), a chronic ulcer, a post-surgical incision, an amputation site(before or after the amputation procedure), a cancerous lesion, ordamaged tissue. Where PPG information is included, the disclosed imagingsystems provide a method to assess pathologies involving changes totissue blood flow and pulse rate including: tissue perfusion;cardiovascular health; wounds such as ulcers; peripheral arterialdisease, and respiratory health.

At block 1520, the data captured by the multispectral multi-apertureimaging system 1513 can be processed into a multispectral datacube 1525having a number of different wavelengths 1523, and, optionally, a numberof different images at the same wavelength corresponding to differenttimes (PPG data 1522). For example, the image processor 1120 can beconfigured by the datacube generation module 1140 to generate themultispectral datacube 1525 according to the workflow 1300. Someimplementations may also associated depth values with various pointsalong the spatial dimensions, as described above.

At block 1530, the multispectral datacube 1525 can be analyzed as inputdata 1525 into a machine learning model 1532 to generate a classifiedmapping 1535 of the imaged tissue. The classified mapping can assigneach pixel in the image data (which, after registration, representspecific points on the imaged object 1511) to a certain tissueclassification, or to a certain healing potential score. The differentclassifications and scores can be represented using visually distinctcolors or patterns in the output classified image. Thus, even though anumber of images are captured of the object 1511, the output can be asingle image of the object (e.g., a typical RGB image) overlaid withvisual representations of pixel-wise classification.

The machine learning model 1532 can be an artificial neural network insome implementations. Artificial neural networks are artificial in thesense that they are computational entities, inspired by biologicalneural networks but modified for implementation by computing devices.Artificial neural networks are used to model complex relationshipsbetween inputs and outputs or to find patterns in data, where thedependency between the inputs and the outputs cannot be easilyascertained. A neural network typically includes an input layer, one ormore intermediate (“hidden”) layers, and an output layer, with eachlayer including a number of nodes. The number of nodes can vary betweenlayers. A neural network is considered “deep” when it includes two ormore hidden layers. The nodes in each layer connect to some or all nodesin the subsequent layer and the weights of these connections aretypically learnt from data during the training process, for examplethrough backpropagation in which the network parameters are tuned toproduce expected outputs given corresponding inputs in labeled trainingdata. Thus, an artificial neural network is an adaptive system that isconfigured to change its structure (e.g., the connection configurationand/or weights) based on information that flows through the networkduring training, and the weights of the hidden layers can be consideredas an encoding of meaningful patterns in the data.

A fully connected neural network is one in which each node in the inputlayer is connected to each node in the subsequent layer (the firsthidden layer), each node in that first hidden layer is connected in turnto each node in the subsequent hidden layer, and so on until each nodein the final hidden layer is connected to each node in the output layer.

A CNN is a type of artificial neural network, and like the artificialneural network described above, a CNN is made up of nodes and haslearnable weights. However, the layers of a CNN can have nodes arrangedin three dimensions: width, height, and depth, corresponding to the 2×2array of pixel values in each video frame (e.g., the width and height)and to the number of video frames in the sequence (e.g., the depth). Thenodes of a layer may only be locally connected to a small region of thewidth and height layer before it, called a receptive field. The hiddenlayer weights can take the form of a convolutional filter applied to thereceptive field. In some embodiments, the convolutional filters can betwo-dimensional, and thus, convolutions with the same filter can berepeated for each frame (or convolved transformation of an image) in theinput volume or for designated subset of the frames. In otherembodiments, the convolutional filters can be three-dimensional and thusextend through the full depth of nodes of the input volume. The nodes ineach convolutional layer of a CNN can share weights such that theconvolutional filter of a given layer is replicated across the entirewidth and height of the input volume (e.g., across an entire frame),reducing the overall number of trainable weights and increasingapplicability of the CNN to data sets outside of the training data.Values of a layer may be pooled to reduce the number of computations ina subsequent layer (e.g., values representing certain pixels may bepassed forward while others are discarded), and further along the depthof the CNN pool masks may reintroduce any discarded values to return thenumber of data points to the previous size. A number of layers,optionally with some being fully connected, can be stacked to form theCNN architecture.

During training, an artificial neural network can be exposed to pairs inits training data and can modify its parameters to be able to predictthe output of a pair when provided with the input. For example, thetraining data can include multispectral datacubes (the input) andclassified mappings (the expected output) that have been labeled, forexample by a clinician who has designated areas of the wound thatcorrespond to certain clinical states, and/or with healing (1) ornon-healing (0) labels sometime after initial imaging of the wound whenactual healing is known. Other implementations of the machine learningmodel 1532 can be trained to make other types of predictions, forexample the likelihood of a wound healing to a particular percentagearea reduction over a specified time period (e.g., at least 50% areareduction within 30 days) or wound states such as, hemostasis,inflammation, pathogen colonization, proliferation, remodeling orhealthy skin categories. Some implementations may also incorporatepatient metrics into the input data to further increase classificationaccuracy, or may segment training data based on patient metrics to traindifferent instances of the machine learning model 1532 for use withother patients having those same patient metrics. Patient metrics caninclude textual information or medical history or aspects thereofdescribing characteristics of the patient or the patient's healthstatus, for example the area of a wound, lesion, or ulcer, the BMI ofthe patient, the diabetic status of the patient, the existence ofperipheral vascular disease or chronic inflammation in the patient, thenumber of other wounds the patient has or has had, whether the patientis or has recently taken immunosuppressant drugs (e.g., chemotherapy) orother drugs that positively or adversely affect wound healing rate,HbA1c, chronic kidney failure stage IV, type II vs type I diabetes,chronic anemia, asthma, drug use, smoking status, diabetic neuropathy,deep vein thrombosis, previous myocardial infarction, transient ischemicattacks, or sleep apnea or any combination thereof. These metrics can beconverted into a vector representation through appropriate processing,for example through word-to-vec embeddings, a vector having binaryvalues representing whether the patient does or does not have thepatient metric (e.g., does or does not have type I diabetes), ornumerical values representing a degree to which the patient has eachpatient metric.

At block 1540, the classified mapping 1535 can be output to a user. Inthis example, the classified mapping 1535 uses a first color 1541 todenote pixels classified according to a first state and uses a secondcolor 1542 to denote pixels classified according to a second state. Theclassification and resulting classified mapping 1535 may excludebackground pixels, for example based on object recognition, backgroundcolor identification, and/or depth values. As illustrated, someimplementations of the multispectral multi-aperture imaging system 1513can project the classified mapping 1535 back on to the tissue site. Thiscan be particularly beneficial when the classified mapping includes avisual representation of a recommended margin and/or depth of excision.

These methods and systems may provide assistance to clinicians andsurgeons in the process of dermal wound management, such as burnexcision, amputation level, lesion removal, and wound triage decisions.Alternatives described herein can be used to identify and/or classifythe severity of decubitus ulcers, hyperaemia, limb deterioration,Raynaud's Phenomenon, scleroderma, chronic wounds, abrasions,lacerations, hemorrhaging, rupture injuries, punctures, penetratingwounds, skin cancers, such as basal cell carcinoma, squamous cellcarcinoma, melanoma, actinic keratosis, or any type of tissue change,wherein the nature and quality of the tissue differs from a normalstate. The devices described herein may also be used to monitor healthytissue, facilitate and improve wound treatment procedures, for exampleallowing for a faster and more refined approach for determining themargin for debridement, and evaluate the progress of recovery from awound or disease, especially after a treatment has been applied. In somealternatives described herein, devices are provided that allow for theidentification of healthy tissue adjacent to wounded tissue, thedetermination of an excision margin and/or depth, the monitoring of therecovery process after implantation of a prosthetic, such as a leftventricular assist device, the evaluation of the viability of a tissuegraft or regenerative cell implant, or the monitoring of surgicalrecovery, especially after reconstructive procedures. Moreover,alternatives described herein may be used to evaluate the change in awound or the generation of healthy tissue after a wound, in particular,after introduction of a therapeutic agent, such as a steroid, hepatocytegrowth factor, fibroblast growth factor, an antibiotic, or regenerativecells, such as an isolated or concentrated cell population thatcomprises stem cells, endothelial cells and/or endothelial precursorcells.

Overview of Example Distributed Computing Environment

FIG. 17 depicts a schematic block diagram of an example distributedcomputing system 1600 including a multispectral multi-aperture imagingsystem 1605, which can be any of the multispectral multi-apertureimaging systems of FIGS. 3A-10B and 12 . As depicted the datacubeanalysis servers 1615 may include one or more computers, perhapsarranged in a cluster of servers or as a server farm. The memory andprocessors that make up these computers may be located within onecomputer or distributed throughout many computers (including computersthat are remote from one another).

The multispectral multi-aperture imaging system 1605 can includenetworking hardware (e.g., a wireless Internet, satellite, Bluetooth, orother transceiver) for communicating over the network 1610 with userdevices 1620 and datacube analysis servers 1615. For example, in someimplementations the processor of the multispectral multi-apertureimaging system 1605 may be configured to control image capture, and thensend raw data to the datacube analysis servers 1615. Otherimplementations of the processor of the multispectral multi-apertureimaging system 1605 may be configured to control image capture andperform spectral unmixing and disparity correction to generate amultispectral datacube, which is then sent to the datacube analysisservers 1615. Some implementations can perform full processing andanalysis locally on the multispectral multi-aperture imaging system1605, and may send the multispectral datacube and resulting analysis tothe datacube analysis servers 1615 for aggregate analysis and/or use intraining or retraining machine learning models. As such, the datacubeanalysis servers 1615 may provide updated machine learning models to themultispectral multi-aperture imaging system 1605. The processing load ofgenerating the end result of analyzing the multispectral datacube can besplit between the multi-aperture imaging system 1605 and the datacubeanalysis servers 1615 in various ways, depending upon the processingpower of the multi-aperture imaging system 1605.

The network 1610 can comprise any appropriate network, including anintranet, the Internet, a cellular network, a local area network or anyother such network or combination thereof. User devices 1620 can includeany network-equipped computing device, for example desktop computers,laptops, smartphones, tablets, e-readers, or gaming consoles, and thelike. For example, results (e.g., classified images) determined by themulti-aperture imaging system 1605 and the datacube analysis servers1615 may be sent to designated user devices of patients, doctors,hospital information systems storing electronic patient medical records,and/or centralized health databases (e.g., of the Center for DiseaseControl) in tissue classification scenarios.

Example Implementation Outcomes

Background: Morbidity and mortality resulting from burns is a majorproblem for wounded warfighters and their care providers. The incidenceof burns among combat casualties has historically been 5-20% withapproximately 20% of these casualties requiring complex burn surgery atthe US Army Institute of Surgical Research (ISR) burn center orequivalent. Burn surgery requires specialized training and is thereforeprovided by ISR staff rather than US Military Hospital staff. Thelimited number of burn specialists leads to high logistical complexityof providing care to burned soldiers. Therefore, a new objective methodof pre-operative and intra-operative detection of burn depth couldenable a broader pool of medical staff, including non-ISR personnel, tobe enlisted in the care of patients with burn wounds sustained incombat. This augmented pool of care providers could then be leveraged toprovide more complex burn care further forward in the role of care ofwarfighters with burn wounds.

In order to begin addressing this need, a novel cart-based imagingdevice that uses multispectral imaging (MSI) and artificial intelligence(AI) algorithms to aide in the preoperative determination of burnhealing potential has been developed. This device acquires images from awide area of tissue (e.g., 5.9×7.9 in2) in a short amount of time (e.g.,within 6, 5, 4, 3, 2, or 1 second(s)) and does not require the injectionof imaging contrast agents. This study based in a civilian populationshows that the accuracy of this device in determining burn healingpotential exceeds clinical judgement by burn experts (e.g., 70-80%).

Methods: Civilian subjects with various burn severities were imagedwithin 72 hours of their burn injury and then at several subsequent timepoints up to 7 days post-burn. True burn severity in each image wasdetermined using either 3-week healing assessments or punch biopsies.The accuracy of the device to identify and differentiate healing andnon-healing burn tissue in first, second, and third degree burn injurieswas analyzed on a per image pixel basis.

Results: Data were collected from 38 civilian subjects with 58 totalburns and 393 images. The AI algorithm achieved 87.5% sensitivity and90.7% specificity in predicting non-healing burn tissue.

Conclusions: The device and its AI algorithm demonstrated accuracy indetermining burn healing potential that exceeds the accuracy of clinicaljudgement of burn experts. Future work is focused on redesigning thedevice for portability and evaluating its use in an intra-operativesetting. Design changes for portability include reducing the size of thedevice to a portable system, increasing the field of view, reducingacquisition time to a single snapshot, and evaluating the device for usein an intra-operative setting using a porcine model. These developmentshave been implemented in a benchtop MSI subsystem that shows equivalencyin basic imaging tests.

Additional Illuminants for Image Registration

In various embodiments, one or more additional illuminants may be usedin conjunction with any of the embodiments disclosed herein in order toimprove the accuracy of image registration. FIG. 21 illustrates anexample embodiment of a multi aperture spectral imager 2100 including aprojector 2105. In some embodiments, the projector 2105 or othersuitable illuminant may be, for example, one of the illuminants 1165described with reference to FIG. 12 above. In embodiments including anadditional illuminant such as a projector 2105 for registration, themethod may further include an additional exposure. The additionalilluminant such as the projector 2105 can project, into the field ofview of the imager 2100, one or more points, fringes, grids, randomspeckle, or any other suitable spatial pattern in a spectral band,multiple spectral bands, or in a broad band, that are individually orcumulatively visible in all cameras of the imager 2100. For example, theprojector 2105 may project light of the shared or common channel,broadband illumination, or cumulatively visible illumination that can beused to confirm the accuracy of the registration of the image calculatedbased on the aforementioned common band approach. As used herein,“cumulatively visible illumination” refers to a plurality of wavelengthsselected such that the pattern is transduced by each of the imagesensors in the multi-spectral imaging system. For example, cumulativelyvisible illumination may include a plurality of wavelengths such thatevery channel transduces at least one of the plurality of wavelengths,even if none of the plurality of wavelengths is common to all channels.In some embodiments, the type of pattern projected by the projector 2105may be selected based on the number of apertures in which the patternwill be imaged. For example, if the pattern will be seen by only oneaperture, the pattern may preferably by relatively dense (e.g., may havea relatively narrow autocorrelation such as on the order of 1-10 pixels,20 pixels, less than 50 pixels, less than 100 pixels, etc.), while lessdense or less narrowly autocorrelated patterns may be useful where thepattern will be imaged by a plurality of apertures. In some embodiments,the additional exposure that is captured with the projected spatialpattern is included in the calculation of disparity in order to improvethe accuracy of the registration compared to embodiments without theexposure captured with a projected spatial pattern. In some embodiments,the additional illuminant projects, into the field of view of theimager, fringes in a spectral band, multiple spectral bands, or in abroad band, that are individually or cumulatively visible in allcameras, such as in the shared or common channel, or broadbandillumination which can be used to improve the registration of imagesbased on the phase of fringes. In some embodiments, the additionalilluminant projects, into the field of view of the imager, a pluralityof unique spatial arrangement of dots, grids, and/or speckle in aspectral band, multiple spectral bands, or in a broad band, that areindividually or cumulatively visible in all cameras, such as in theshared or common channel, or broadband illumination which can be used toimprove the registration of images. In some embodiments, the methodfurther includes an additional sensor with a single aperture or aplurality of apertures, which can detect the shape of the object orobjects in the field of view. For example, the sensor may use LIDAR,light field, or ultrasound techniques, to further improve the accuracyof registration of the images using the aforementioned common bandapproach. This additional sensor may be a single aperture or amulti-aperture sensor, sensitive to light-field information, or it maybe sensitive to other signals, such as ultrasound or pulsed lasers.

Machine Learning Implementations for Wound Assessment, HealingPrediction, and Treatment

Example embodiments of machine learning systems and methods for woundassessment, healing prediction, and therapy will now be described. Anyof the various imaging devices, systems, methods, techniques, andalgorithms described herein may be applied in the field of wound imagingand analysis. The following implementations may include the acquisitionof one or more images of a wound in one or more known wavelength bands,and may include, based on the one or more images, any one or more thefollowing: segmentation of the image into a wound portion and anon-wound portion of the image, prediction of percent area reduction ofthe wound after a predetermined time period, prediction of healingpotentional of individual sections of the wound after a predeterminedtime period, display of a visual representation associated with any suchsegmentation or prediction, indication of a selection between a standardwound care therapy and an advanced wound care therapy, and the like.

In various embodiments, a wound assessment system or a clinician candetermine an appropriate level of wound care therapy based on theresults of the machine learning algorithms disclosed herein. Forexample, if an output of a wound healing prediction system indicatesthat an imaged wound will close by more than 50% within 30 days, thesystem can apply or inform a health care practitioner or patient toapply a standard of care therapy; if the output indicates that the woundwill not close by more than 50% in 30 days, the system can apply orinform the health care practitioner or patient to use one or moreadvanced wound care therapies.

Under existing wound treatment, a wound such as a diabetic foot ulcer(DFU) may initially receive one or more standard wound care therapiesfor the initial 30 days of treatment, such as Standard of Care (SOC)therapy as defined by the Centers for Medicare and Medicaid. As oneexample of a standard wound care regimen, SOC therapy can include one ormore of: optimization of nutritional status; debridement by any means toremove devitalized tissue; maintenance of a clean, moist bed ofgranulation tissue with appropriate moist dressings; necessary treatmentto resolve any infection that may be present; addressing anydeficiencies in vascular perfusion to the extremity with the DFU;offloading of pressure from the DFU; and appropriate glucose control.During this initial period of 30 days of SOC therapy, measurable signsof DFU healing are defined as: decrease in DFU size (either woundsurface area or wound volume), decrease in amount of DFU exudate, anddecrease in amount of necrotic tissue within the DFU. An exampleprogression of a healing DFU is illustrated in FIG. 22 .

If healing is not observed during this initial period of 30 days of SOCtherapy, Advanced Wound Care (AWC) therapies are generally indicated.The Centers for Medicare and Medicaid have no summary or definition ofAWC therapies but are considered to be any therapy outside of SOCtherapy as defined above. AWC therapies are an area of intense researchand innovation with near-constant introduction of new options to be usedin clinical practice. Therefore, coverage of AWC therapies aredetermined on an individual basis and a treatment considered AWC may notbe covered for reimbursement for some patients. Based on thisunderstanding, AWC therapies include, but are not limited to, any one ormore of: hyperbaric oxygen therapy; negative-pressure wound therapy;bioengineered skin substitutes; synthetic growth factors; extracellularmatrix proteins; matrix metalloproteinase modulators; and electricalstimulation therapy. An example progression of a non-healing DFU isillustrated in FIG. 23 .

In various embodiments, wound assessment and/or healing predictionsdescribed herein may be accomplished based on one or more images of thewound, either alone or based on a combination of both patient healthdata (e.g., one or more health metric values, clinical features, etc.)and images of the wound. The described techniques can capture singleimages or a set of multispectral images (MSI) of a patient tissue siteincluding an ulcer or other wound, process the image(s) using a machinelearning system as described herein, and output one or more predictedhealing parameters. A variety of healing parameters may be predicted bythe present technology. By way of non-limiting example, some predictedhealing parameters may include (1) a binary yes/no regarding whether theulcer will heal to greater than 50% area reduction (or another thresholdpercentage, as desired according to clinical standards) within a periodof 30 days (or another time period, as desired according to clinicalstandards); (2) a percentage likelihood that the ulcer will heal togreater than 50% area reduction (or another threshold percentage, asdesired according to clinical standards) within a period of 30 days (oranother time period, as desired according to clinical standards); or (3)a prediction regarding the actual area reduction that is expected within30 days (or another time period, as desired according to clinicalstandards) due to healing of the ulcer. In further examples, systems ofthe present technology may provide a binary yes/no or a percentagelikelihood of healing with regard to smaller portions of a wound, suchas for individual pixels or subsets of pixels of a wound image, with theyes/no or percentage likelihood indicating whether each individualportion of the wound is likely to be healing tissue or non-healingtissue following the predetermined time period.

FIG. 24 presents one example approach to providing such healingpredictions. As illustrated, an image of a wound, or a set ofmultispectral images of the wound captured at different wavelengths,either at different times or simultaneously using a multispectral imagesensor, may be used to provide both the input and output values to aneural network such as an autoencoder neural network, which is a type ofartificial neural network as described in greater detail below. Thistype of neural network is able to generate a reduced featurerepresentation of the input, here a reduced number of values (e.g.,numerical values) representing the pixel values in the input image(s).This can in turn be provided to a machine learning classifier, forexample a fully connected feedforward artificial neural network or thesystem shown in FIG. 25 , in order to output a healing prediction forthe imaged ulcer or other wound.

FIG. 25 presents another approach to providing such healing predictions.As illustrated, an image (or set of multispectral images captured atdifferent wavelengths, either at different times or simultaneously usinga multispectral image sensor) is provided as input into a neural networksuch as a convolutional neural network (“CNN”). The CNN takes thistwo-dimensional (“2D”) array of pixel values (e.g., values along boththe height and width of the image sensor used to capture the image data)and outputs a one-dimensional (“1D) representation of the image. Thesevalues can represent classifications of each pixel in the input image,for example according to one or more physiological states pertaining toulcers or other wounds.

As shown in FIG. 25 , a patient metric data repository can store othertypes of information about the patient, referred to herein as patientmetrics, clinical variables, or health metric values. Patient metricscan include textual information describing characteristics of thepatient, for example, the area of the ulcer, the body mass index (BMI)of the patient, the number of other wounds the patient has or has had,diabetic status, whether the patient is or has recently takenimmunosuppressant drugs (e.g., chemotherapy) or other drugs thatpositively or adversely affect wound healing rate, HbA1c, chronic kidneyfailure stage IV, type II vs. type I diabetes, chronic anemia, asthma,drug use, smoking status, diabetic neuropathy, deep vein thrombosis,previous myocardial infarction, transient ischemic attacks, or sleepapnea or any combination thereof. However, a variety of other metricsmay be used. A number of example metrics are provided in Table 1 below.

TABLE 1 Example clinical variables for wound image analysis VariableDescription General demographics Age Age of the patient. Gender Genderof the patient. Race Race of the patient. Ethnicity Ethnicity of thepatient. Height Height of patient. Weight Weight of patient. BMIBody-mass-index of patient. DFU History Prior DFU Number of prior DFUsPrior DFU Healing Healing rates and times of prior DFUs Prior DFULocation Locations of prior DFUs Prior DFU Size Size of prior DFUsCurrent DFU Number of current DFUs Current DFU Location Location ofcurrent DFUs Current DFU Size Size of current DFUs DFU TreatmentDuration Duration of current DFUs prior to seeking treatment. DFUTreatment Prior Prior treatments and duration of treatments performed oncurrent DFUs. DFU Treatment Planned Planned treatment of current DFUs.DFU Current Healing Healing response of current DFUs to priortreatments. DFU Stage Class and/or stage of current DFUs using widelyaccepted grading schemes including the Wagner classification. DFU CauseCausative event of current DFUs. DFU Infection Infection status ofcurrent DFUs. DFU Infection Cause Causative microorganism of current orprior DFU infections. DFU CFU Colony-forming-unit count of current DFUinfections. Compliance Socio-Economic Socio-economic status of thepatient. Mal-Compliance History of mal-compliance with healthcare.Functional Status Current functional status of patient. EndocrineDiabetes Mellitus Duration Duration of diabetes mellitus diagnosis forpatient. Diabetes Mellitus Type Type of diabetes mellitus diagnosed inpatient. Hemoglobin A1C % Current or most recent hemoglobin A1C % valuefor patient. Serum Glucose Current or most recent serum glucose valuefor patient. Diabetes Mellitus Current glycemic control mediations,including insulin, Medications taken by patient. Diabetes MellitusNeuropathy Presence of diabetic peripheral neuropathy. Diabetes MellitusFoot Presence of foot malformations due to diabetes mellitus and/orperipheral neuropathy. Steroids Current or prior use of glucocorticoidmedications. Other Endocrine Current or prior medical diagnosis directlyor indirectly altering endocrine or metabolic systems. Other EndocrineMedications Current or prior medications directly or indirectly alteringendocrine or metabolic systems. Cardiovascular Peripheral VascularDisease Presence of peripheral vascular disease. Ankle Brachial IndexCurrent or most recent ankle-brachial index for available extremities.Endovascular Procedures History and locations of prior extremityendovascular angioplasty, stenting, or other procedure to treatperipheral vascular disease. Bypass Procedures History and locations ofprior extremity surgical bypass procedures to treat peripheral vasculardisease. Bypass Healing Healing rates and times of prior extremitysurgical bypass procedures. Endocarditis Current or prior endocarditis.Cerebrovascular Accidents Current or prior cerebrovascular accidents(strokes). Neurological Deficits Neurological deficits remaining fromprior cerebrovascular accidents. Anemia Current or prior diagnosis ofanemia Hemoglobin/Hematocrit Current or most recent serum hemoglobin andhematocrit values. Venous Thrombosis Current or prior diagnosis of deepor superficial venous thrombosis. Anticoagulation Medications Current orprior use of anti-coagulation medications. Atrial Fibrillation Currentor prior diagnosis of atrial fibrillation. Heart Failure Current orprior diagnosis of heart failure of any type. Heart Attack Current orprior diagnosis of myocardial infarction (heart attack). CardiacStenting Prior history of cardiac stenting procedures. PlateletMedications Current or prior anti-platelet or otherwiseplatelet-altering medications. Calcium Blocking Current use of calciumblocking medications. Medications Sympathetic Medications Current use ofsympathetic-modifying medications including medications inducingbeta-blockade. Diuretic Medications Current use of diuresis-inducingmedications. Renin Aldosterone Current use of medications altering therenin-aldosterone Medications pathway. Dromotropic Medications Currentuse of medications altering the inotropy, chronotropy, or dromotropy ofthe heart. Lipid Cholesterol Current use of medications altering lipidor cholesterol Medications pathways. Musculoskeletal Other FootDeformities Presence of foot malformations not due to diabetes mellitusand/or peripheral neuropathy. Prior Amputations History and locations ofprior extremity amputations. Prior Amputation Healing Healing rates andtimes of prior amputations. Nutrition Nutrition Deficit Currentdiagnosis of malnutrition or malnourishment of any type includingdeficiencies of calories, fats, proteins, vitamins, and minerals.Nutrition Illness Presence of any medical illness directly or indirectlyaltering nutrition status of any type. Nutrition Markers Serum albumin,pre-albumin, or transferrin values below the normal range set by themeasuring facility. Nutrition Signs Symptoms Presence of physical examfindings or patient history known to be indicative of malnutrition.Nutrition Treatment Current treatment for malnutrition or malnourishmentof any type including enteral nutrition, parenteral nutrition, andsupplementation of calories, fats, proteins, vitamins, and minerals.Infectious Disease Infections Wound Current or prior infections of anywounds. Infections Deep Current or prior diagnosis of deep tissueinfections including osteomyelitis. Infections Cause Main causativemicroorganism for current or prior wound infections. Infections SystemicCurrent diagnosis of infection located anywhere. Antibiotics UsageCurrent or prior use of antibiotic treatment defined as prolonged. RenalChronic Kidney Disease Current or prior diagnosis of chronic kidneydisease. Kidney Disease Stage Stage or severity of chronic kidneydisease. Dialysis Current need for dialysis of any type. CreatinineCurrent or most recent serum creatinine value. Creatinine ClearanceCurrent or most recent creatinine clearance value. Acute Kidney InjuryPresence of acute kidney injury. Ob/Gyn Pregnancy Current pregnancystatus of the patient. Past Pregnancy Pregnancy history of the patient.Menopause Pre- or Post-menopausal status of the patient. HormoneMedications Current hormonal medications taken by patient. Other TobaccoUse Current and prior tobacco use status of patient. Tobacco MethodMethod of current or prior tobacco use. Tobacco Amount Amount of currentor prior tobacco use. Alcohol Use Current and prior alcohol use ofpatient. Alcohol Abuse Current or prior alcohol abuse. Illicit Drug UseCurrent or prior illicit drug use, including marijuana. Cancer StatusCurrent or prior diagnosis of cancers. Cancer Locations Locations andtypes of prior cancers. Cancer Recurrence Presence and location ofcancer recurrences. Chemotherapy Radiation Current or prior treatmentwith chemotherapy medications, radiation therapy, or other treatmentsutilized for cancer. Autoimmune Disorder Current or prior diagnosis ofautoimmune disorders. Autoimmune Treatments Current or prior treatmentsfor autoimmune disorders including immunosuppressants. Transplant StatusPrior history of organ transplant surgery. Transplant MedicationsCurrent or prior medications for management of transplanted organsincluding immunosuppressants. Asthma Current or prior history of asthma.Asthma Treatments Current or prior treatments for asthma includingcorticosteroids. Liver Cirrhosis Current or prior diagnosis of livercirrhosis. MELD Score Current or most recent MELD or MELD XI score.Child-Pugh Score Current or most recent Child-Pugh score. Otherlaboratory values Sodium Current or most recent serum sodium value.Potassium Current or most recent serum potassium value. Chloride Currentor most recent serum chloride value. Bicarbonate Current or most recentserum bicarbonate value. Bilirubin Current or most recent serum totaland direct bilirubin values. Aspartate Transaminase Current or mostrecent serum aspartate transaminase value. Alanine Transaminase Currentor most recent serum alanine transaminase value. Total Protein Currentor most recent serum total protein value. White Blood Cell Current ormost recent serum white blood cell count. Platelet Current or mostrecent serum platelet count. Lactate Current or most recent serumlactate value. Lactate Dehydrogenase Current or most recent serumlactate dehydrogenase value. Calcium Current or most recent serumcalcium value. Magnesium Current or most recent serum magnesium value.Phosphorus Current or most recent serum phosphorus value. ProcalcitoninCurrent or most recent serum procalcitonin value. Other medications

These metrics can be converted into a vector representation throughappropriate processing, for example through word-to-vec embeddings, avector having binary values representing whether the patient does ordoes not have the patient metric (e.g., does or does not have type Idiabetes), or numerical values representing a degree to which thepatient has each patient metric. Various embodiments can use any one ofthese patient metrics or a combination of some or all of the patientmetrics to improve the accuracy of predicted healing parametersgenerated by the systems and methods of the present technology. In anexample trial, it was determined that image data taken during theinitial clinical visit for a DFU, analyzed alone without consideringclinical variables, could accurately predict percent area reduction ofthe DFU with approximately 67% accuracy. Predictions based on patientmedical history alone were approximately 76% accurate, with the mostimportant features being: wound area, BMI, number of previous wounds,HbA1c, chronic kidney failure stage IV, type II vs type I diabetes,chronic anemia, asthma, drug use, smoking status, diabetic neuropathy,deep vein thrombosis, previous myocardial infarction, transient ischemicattacks, and sleep apnea. When combining these medical variables withimage data we observed an increase in prediction accuracy toapproximately 78%.

In one example embodiment as shown in FIG. 25 , the 1D representation ofthe image data can be concatenated with the vector representation of thepatient metrics. This concatenated value can then be provided as aninput into a fully connected neural network, which outputs a predictedhealing parameter.

The system shown in FIG. 25 can be considered as a single machinelearning system having multiple machine learning models as well as thepatient metric vector generator. In some embodiments, this entire systemcan be trained in an end-to-end fashion such that the CNN and fullyconnected network tune their parameters through backpropagation in orderto be able to generate predicted healing parameters from input images,with the patient metric vector added to the values passed between theCNN and the fully connected network.

Example Machine Learning Models

Artificial neural networks are artificial in the sense that they arecomputational entities, inspired by biological neural networks butmodified for implementation by computing devices. Artificial neuralnetworks are used to model complex relationships between inputs andoutputs or to find patterns in data, where the dependency between theinputs and the outputs cannot be easily ascertained. A neural networktypically includes an input layer, one or more intermediate (“hidden”)layers, and an output layer, with each layer including a number ofnodes. The number of nodes can vary between layers. A neural network isconsidered “deep” when it includes two or more hidden layers. The nodesin each layer connect to some or all nodes in the subsequent layer andthe weights of these connections are typically learned based on trainingdata during the training process, for example, through backpropagationin which the network parameters are tuned to produce expected outputsgiven corresponding inputs in labeled training data. Thus, an artificialneural network may be an adaptive system that is configured to changeits structure (e.g., the connection configuration and/or weights) basedon information that flows through the network during training, and theweights of the hidden layers can be considered as an encoding ofmeaningful patterns in the data.

A fully connected neural network is one in which each node in the inputlayer is connected to each node in the subsequent layer (the firsthidden layer), each node in that first hidden layer is connected in turnto each node in the subsequent hidden layer, and so on until each nodein the final hidden layer is connected to each node in the output layer.

Autoencoders are neural networks that include an encoder and a decoder.The goal of certain autoencoders is to compress the input data with theencoder, then decompress this encoded data with the decoder such thatthe output is a good/perfect reconstruction of the original input data.Example autoencoder neural networks described herein, such as theautoencoder neural network illustrated in FIG. 24 , can take the imagepixel values of an image of a wound (e.g., structured in vector ormatrix form) as inputs into its input layer. The subsequent one or morelayers, or “encoder layers,” encode this information by lowering itsdimensionality (e.g., by representing the input using fewer dimensionsthan its original n-dimensions), and the additional one or more hiddenlayers subsequent to the encoder layers (“decoder layers”) decode thisinformation to generate an output feature vector at the output layer. Anexample training process for the autoencoder neural network can beunsupervised, in that the autoencoder learns the parameters of itshidden layers that produce the same output as the provided input. Assuch, the number of nodes in the input and output layers are typicallythe same. The dimensionality reduction allows the autoencoder neuralnetwork to learn the most salient features of the input images, wherethe innermost layer (or another inner layer) of the autoencoderrepresents a “feature reduction” version of the input. In some examples,this can serve to reduce an image having, for example, approximately 1million pixels (where each pixel value can be considered as a separatefeature of the image) to a feature set of around 50 values. Thisreduced-dimensionality representation of the images can be used byanother machine learning model, for example, the classifier of FIG. 25or a suitable CNN or other neural network, in order to output apredicted healing parameter.

A CNN is a type of artificial neural network, and like the artificialneural networks described above, a CNN is made up of nodes and haslearnable weights between nodes. However, the layers of a CNN can havenodes arranged in three dimensions: width, height, and depth,corresponding to the 2×2 array of pixel values in each image frame(e.g., the width and height) and to the number of image frames in asequence of images (e.g., the depth). In some embodiments, the nodes ofa layer may only be locally connected to a small region of the width andheight of the preceding layer, called a receptive field. The hiddenlayer weights can take the form of a convolutional filter applied to thereceptive field. In some embodiments, the convolutional filters can betwo-dimensional, and thus, convolutions with the same filter can berepeated for each frame (or convolved transformation of an image) in theinput volume or for designated subset of the frames. In otherembodiments, the convolutional filters can be three-dimensional and thusextend through the full depth of nodes of the input volume. The nodes ineach convolutional layer of a CNN can share weights such that theconvolutional filter of a given layer is replicated across the entirewidth and height of the input volume (e.g., across an entire frame),reducing the overall number of trainable weights and increasingapplicability of the CNN to data sets outside of the training data.Values of a layer may be pooled to reduce the number of computations ina subsequent layer (e.g., values representing certain pixels may bepassed forward while others are discarded), and further along the depthof the CNN pool masks may reintroduce any discarded values to return thenumber of data points to the previous size. A number of layers,optionally with some being fully connected, can be stacked to form theCNN architecture. During training, an artificial neural network can beexposed to pairs in its training data and can modify its parameters tobe able to predict the output of a pair when provided with the input.

Artificial intelligence describes computerized systems that can performtasks typically considered to require human intelligence. Here, thedisclosed artificial intelligence systems can perform image (and otherdata) analysis that, without the disclosed technology, may otherwiserequire the skill and intelligence of a human physician. Beneficially,the disclosed artificial intelligence systems can make such predictionsupon an initial patient visit rather than requiring a wait time of 30days to assess wound healing.

The capability to learn is an important aspect of intelligence, as asystem without this capability generally cannot become more intelligentfrom experience. Machine learning is a field of computer science thatgives computers the ability to learn without being explicitlyprogrammed, for example, enabling artificial intelligence systems tolearn complex tasks or adapt to changing environments. The disclosedmachine learning systems can learn to determine wound healing potentialthrough being exposed to large volumes of labeled training data. Throughthis machine learning, the disclosed artificially intelligent systemscan learn new relationships between the appearances of wounds (ascaptured in image data such as MSI) and the healing potential of thewound.

The disclosed artificially intelligent machine learning systems includecomputer hardware one or more memories and one or more processors, forexample, as described with reference to the various imaging systemsherein. Any of the machine learning systems and/or methods of thepresent technology may be implemented on or in communication withprocessors and/or memory of the various imaging systems and devices ofthe present disclosure.

Example Multispectal DFU Imaging Implementation

In an example application of the machine learning systems and methodsdisclosed herein, machine learning algorithms consistent with thosedescribed above were used to predict the percent area reduction (PAR) ofan imaged wound at day 30, following imaging on day 0. To accomplishthis prediction, a machine learning algorithm was trained to take MSIdata and clinical variables as inputs and to output a scalar valuerepresenting the predicted PAR. After 30 days, each wound was evaluatedto measure its true PAR. The predicted PAR was compared to the true PARmeasured during a 30-day healing assessment conducted on the wound. Theperformance of the algorithm was scored using a coefficient ofdetermination (R²).

The machine learning algorithm for this example application was abagging ensemble of decision tree classifers, fit using data from adatabase of DFU image. Other suitable classifier ensembles may equallybe implemented, such as the XGBoost algorithm or the like. The databaseof DFU images contained 29 individual images of diabetic foot ulcersobtained from 15 subjects. For each image, the true PAR measured at day30 was known. Algorithm training was conducted using the leave-one-outcross-validation (LOOCV) procedure. The R² score was computed aftercombining the predicted results on the test image from each fold ofLOOCV.

The MSI data consisted of 8 channels of 2D images, where each of the 8channels represented the diffuse reflectance of light from the tissue ata specific wavelength filter. The field of view of each channel was 15cm×20 cm with a resolution of 1044 pixels×1408 pixels. The 8 wavelengthbands included: 420 nm±20 nm; 525 nm±35 nm; 581 nm±20 nm; 620 nm±20 nm;660 nm±20 nm; 726 nm±41 nm; 820 nm±20 nm; and 855 nm±30 nm, wherein “±”represents the full width at half maximum of each spectral channel. The8 wavelength bands are sillustrated in FIG. 26 . From each channel, thefollowing quantitative features were computed: the mean of all pixelvalues, the median of all pixel values, and the standard deviation ofall pixel values.

Additionally, from each subject, the following clinical variables wereobtained: age, level of chronic kidney disease, the length of the DFU atday 0, and the width of the DFU at day 0.

Separate algorithms were generated using features extracted from allpossible combinations of the 8 channels (wavelength bands) in the MSIdata cube using 1 channel to 8 channels, totaling C₁(8)+C₂ (8)+ . . .+C₈ (8)=255 different feature sets. The R² values from each combinationwere calculated and ordered from smallest to largest. The 95% confidenceinterval of the R² value was computed from the prediction results of thealgorithm trained on each feature set. To determine if a feature setcould provide an improvement over random chance, feature sets wereidentified wherein the value of 0.0 was not contained within the 95% CIof the results of the algorithm trained on that feature set.Additionally, the same analysis was performed an additional 255 timeswith the inclusion of all the clinical variables in every feature-set.In order to determine whether the clinical variables had an impact onthe performance of the algorithm, the mean R² value from the 255algorithms trained using the clinical variables was compared to the 255algorithms trained without the clinical variables using a t-test. Theresults of the analysis are illustrated in Tables 2 and 3, below. Table2 illustrates the performance of feature sets including only image datawithout including clinical variables.

TABLE 2 Top performing algorithms developed on feature sets that did notinclude clinical data Lower Upper Rank Feature Set R² 95% CI 95% CI 1[420, 726, 855] 0.53 0.34 0.72 2 [420, 525, 660, 726, 820] 0.51 0.400.63 3 [420, 581, 660, 726, 820, 855] 0.48 0.37 0.58 4 [420, 525, 581,620, 660, 726, 855] 0.48 0.38 0.57 5 [660, 726, 855] 0.46 0.26 0.67 . .. . . . . . . . . . . . . 251 [581] 0.11 −0.07 0.30 252 [525] 0.11 −0.070.29 253 [620] 0.10 −0.08 0.28 254 [820] 0.05 −0.09 0.18 255 [726] 0.04−0.08 0.16

As shown in Table 2, among feature sets that did not include theclinical features, the top performing feature-set contained only 3 ofthe 8 possible channels in the MSI data. It was observed that the 726 nmwavelength band appears in all top 5 feature sets. Only one wavelengthband appears in each of the bottom five feature sets. It was furtherobserved that although the 726 nm wavelength band appeared in each ofthe top 5 feature sets, the 726 nm wavelength band performed the worstwhen used alone. Table 3 below illustrates the performance of featuresets including image data as well as the clinical variables of age,level of chronic kidney disease, length of the DFU at day 0, and widthof the DFU at day 0.

TABLE 3 Top performing algorithms developed on feature sets thatincluded clinical data Lower Upper Rank Feature Set R² 95% CI 95% CI 1[CFs, 420, 525, 581, 620, 0.56 0.47 0.65 660, 726, 820, 855] 2 [CFs,420, 581, 660, 820, 855] 0.55 0.44 0.65 3 [CFs, 420, 581, 660, 726, 855]0.52 0.41 0.63 4 [CFs, 660, 726, 855] 0.49 0.31 0.67 5 [CFs, 525, 581,620, 820, 855] 0.46 0.24 0.68 . . . . . . . . . . . . . . . 251 [CFs,420, 620, 660] 0.17 −0.04 0.37 252 [CFs, 820] 0.16 −0.04 0.36 253 [CFs,420, 820, 855] 0.15 −0.02 0.32 254 [CFs, 620] 0.14 −0.04 0.33 255 [CFs,420] 0.11 −0.07 0.29

From feature sets that did include the clinical variables, the topperforming feature set contained all 8 of the possible channels in theMSI data. The 855 nm wavelength band appears in all top 5 feature sets.Histograms from models with and without the inclusion of clinicalvariables are illustrated in FIG. 27 , along with vertical linesrepresenting the mean of each distribution.

In comparing the importance of clinical features, it was determinedwhether the mean R² between all features sets without clinical variableswas equal to the mean R² from all feature sets that included clinicalvariables. It was determined that the mean R² from models trained onfeature sets without clinical variables was 0.31, and 0.32 from modeltrained with clinical variables. In computing the t-test for thedifference between means, the p-value was 0.0443. Therefore, it wasdetermind that models trained with clinical variables were significantlymore accurate than models trained without clinical features.

Extraction of Features from Image Data

Although the example application described above extracted mean,standard deviation, and median pixel values, it will be understood thata variety of other features may be extracted from image data for use ingenerating predicted healing parameters. Feature categories includelocal, semi-local, and global features. Local features may representtexture in an image patch, while global features can include contourrepresentations, shape descriptors, and texture features. Global texturefeatures and local features provide different information about theimage because the support over which texture is computed varies. In somecases, global features have the ability to generalize an entire objectwith a single vector. Local features, on the other hand, are computed atmultiple points in the image and are consequently more robust toocclusion and clutter. However, they may require specializedclassification algorithms to handle cases in which there are a variablenumber of feature vectors per image.

Local features may include, for example, scale-invariant featuretransform (SIFT), speeded-up robust features (SURF), features fromaccelerated segment test (FAST), binary robust invariant scalablekeypoints (BRISK), Harris corner detection operator, binary robustindependent elementary features (BRIEF), oriented FAST and rotated BRIEF(ORB), and KAZE features. Semi-local features may include, for example,edges, splines, lines, and moments in small windows. Global features mayinclude, for example, color, Gabor features, wavelet features, Fourierfeatures, texture features (e.g., 1^(st), 2^(nd) and high moments),neural network features from 1D, 2D, and 3D convolutions or hiddenlayers, and principal component analysis (PCA).

Example RGB DFU Imaging Application

As a further example of predicted healing parameter generation, similarMSI methods may be used based on RGB data, such as from a photographicdigital camera. In this scenario, the algorithm can take data from anRGB image, and optionally the subject's medical history or otherclinical variables, and output a predicted healing parameter such as aconditional probability that indicates whether the DFU will respond to30 days of standard wound care therapy. In some embodiments, theconditional probability is the probability that the DFU in question isnon-healing given the input data, x, to a model parameterized by θ;written as: P_(model)(y=“non-healing”|x; θ).

Scoring methods for RGB data may be similar to those for the example MSIapplication described above. In one example, the predicted non-healingregion can be compared to the true non-healing region measured during a30-day healing assessment conducted on a wound such as a DFU. Thiscomparison represents the performance of the algorithm. The methodapplied to perform this comparison may be based on the clinical outcomeof these output images.

In this example application, four outcomes are possible for eachpredicted healing parameter generated by the healing predictionalgorithm. In a True Positive (TP) outcome, the wound demonstrates lessthan 50% area reduction (e.g., the DFU is non-healing), and thealgorithm predicts less than 50% area reduction (e.g., the deviceoutputs a non-healing prediction). In a True Negative (TN) outcome, thewound demonstrates at laest 50% area reduction (e.g., the DFU ishealing), and the algorithm predicts at least 50% area reduction (e.g.,the device outputs a healing prediction). In a False Positive (FP)outcome, the wound demonstrates at laest 50% area reduction, but thealgorithm predicts less than 50% area reduction. In a False Negative(FN) outcome, the wound demonstrates less than 50% area reduction, butthe algorithm predicts at least 50% area reduction. After prediction andassessment of actual healing, these outcomes can be summarized using theperformance metrics of accuracy, sensitivity, and specificity, as shownin Table 4, below.

TABLE 4 Standard performance metrics used to evaluate predictions onimages. Metric Computation Accuracy$\frac{{TP} + {TN}}{{TP} + {FP} + {TN} + {FN}}$ eq. 1 True Positive Rate(TPR; also known as Sensitivity) ${TPR} = \frac{TP}{{TP} + {FN}}$ eq. 2True Negative Rate (TNR; also known as Specificity)${TNR} = \frac{TN}{{TN} + {FP}}$ eq. 3

A database of DFU images was obtained retrospectively and included 149individual images of diabetic foot ulcers from 82 subjects. Of the DFUsin this data set, 69% were considered “healing” because they reached thetarget goal of 50% PAR at day 30. The average wound area was 3.7 cm²,and the median wound area was 0.6 cm².

Color photography images (RGB images) were used as input data to themodels developed. RGB imaged consisted of 3 channels of 2D images, whereeach of the 3 channels represented the diffuse reflectance of light fromthe tissue at the wavelengths utilized in a traditional color camerasensor. Images were captured by a clinician using a portable digitalcamera. The choice of imager, working distance, and field-of-view (FOV)varied between images. Prior to algorithm training, the images weremanually cropped to ensure the ulcer was at the center of the FOV. Aftercropping, the images were interpolated to an image size of 3channels×256 pixels×256 pixels. Maintaining the aspect ratio of theoriginal image was not controlled for during this interpolation step.However, the aspect ratio could be maintained throughout thesepre-processing steps if desired. From each subject, a set of clinicaldata (e.g., clinical variables or health metric values) was alsoobtained including their medical history, prior wounds, and blood work.

Two types of algorithm were developed for this analysis. The goal ofeach algorithm was to initially identify a new representation for theimage data that could be combined with the patient health metrics in atraditional machine learning classification approach. There are manyavailable methods to produce this image representation such as principalcomponent analysis (PCA) or scale-invariant feature transform (SIFT). Inthis example, convolutional neural networks (CNN) were used to transformthe images from a matrix (with dimensions 3 channels×265 pixels×256pixels) to a vector in

^(n). In one example, a separately trained unsupervised approach wasused to compress the images, followed by machine learning to makepredictions on DFU healing. In a second example, an end-to-endsupervised approach was used to predict DFU healing.

In the unsupervised feature extraction approach, an autoencoderalgorithm was used, for example, consistent with the method of FIG. 24 .An example autoencoder is schematically illustrated in FIG. 28 . Theautoencoder included an encoder module and a decoder module. The encodermodule was a 16-layer VGG convolutional network. The 16^(th) layerrepresented the compressed image representation. The decoder module wasa 16-layer VGG network with up-sampling functions added and poolingfunctions eliminated. For each predicted pixel value (in

) of the output of the decoder layer, the loss was computed with meansquare error (MSE) wherein the target values were the pixel values ofthe original image.

The autoencoder was pre-trained using PASCAL visual object classes (VOC)data and fine-tuned using the DFU images in the present data set.Individual images comprising 3 channels×265 pixels×256 pixels (65,536total pixels) were compressed into single vectors of 50 data points.Once trained, the identical encoder-decoder algorithm was used for allimages in the data set.

Upon extraction of the compressed image vector, the compressed imagevector was used as an input to a second supervised machine learningalgorithm. The combination of image features and patient features weretested using a variety of machine learning algorithms, includinglogistic regression, K-nearest neighbors, support vector machine, and avariety of decision tree models. An example supervised machine learningalgorithm, using the compressed image vector and patient clinicalvariables as inputs to predict DFU healing, is schematically illustratedin FIG. 29 . The machine learning algorithm may be one of various knownmachine learning algorithms such as a multi-layer perceptron, quadraticdiscriminant analysis, naïve Bayes, or an ensemble of such algorithms.

The end-to-end machine learning approach, investigated as an alternativeto the unsupervised feature extraction approach described above, isschematically illustrated in FIG. 30 . In the end-to-end approach, the16-layer VGG CNN was modified at the first fully connected layer byconcatenating the patient health metrics data to the image vector. Inthis manner, the encoder module and subsequent machine learningalgorithm could be trained simultaneously. Other methods of includingglobal variables (e.g., patient health metrics or clinical variables) toimprove the performance or alter the purpose of a CNN have beenproposed. The most widely used method is the feature-wise linearmodulation (FiLM) generator. For the supervised machine learningalgorithms, training was performed using a k-fold cross-validationprocedure. The results of each image were computed as one of TruePositive, True Negative, False Positive, or False Negative. Theseresults were summarized using the performance metrics described in Table4, above.

Accuracy of predictions from the unsupervised feature extraction(autoencoder) and machine learning approach of FIGS. 28 and 29 wereobtained using seven different machine learning algorithms and threedifferent input feature combinations, as shown in FIG. 31 . Eachalgorithm was trained using 3-fold cross validation and the averageaccuracy (±95% confidence interval) is reported. Only two algorithmstrained in using this approach exceeded the baseline accuracy. Thebaseline accuracy occurred when a naïve classifier simply predicted allDFUs as healing. The two algorithms that exceeded the baseline werelogistic regression and support vector machines including a combinationof both image data and patient data. The important patient healthmetrics that were predictive of the DFU healing and used to in thesemodels included: wound area; body mass index (BMI); number of previouswounds; hemoglobin A1c (HbA1c); kidney failure; type II vs type Idiabetes; anemia; asthma; drug use; smoking status; diabetic neuropathy;deep vein thrombosis (DVT); or previous myocardial infarction (MI) andcombinations thereof.

Results using the end-to-end machine learning approach of FIG. 30demonstrated performance that was significantly better than thebaseline, as shown in FIG. 32 . While this approach was notsignificantly better than the unsupervised approach, the averageaccuracy was higher than any other method attempted.

Prediction of Healing of a Subset of Wound Area

In further example embodiments, in addition to generating a singlehealing probability for an entire wound, the systems and methods of thepresent technology are further able to predict the area of tissue withinan individual wound that will not be healed after 30 days of standardwound care. To accomplish this output, a machine learning algorithm wastrained to take MSI or RGB data as input and generate predicted healingparameters for portions of the wound (e.g., for individual pixels orsubsets of pixels in a wound image). The present technology can furtherbe trained to output a visual representation such as an image thathighlights the area of ulcer tissue that is not predicted to heal within30 days.

FIG. 33 illustrates an example process of healing prediction andgeneration of a visual representation. As shown in FIG. 33 , a spectraldata cube is obtained, as described elsewhere herein. This data cube ispassed to the machine learning software for processing. Machine learningsoftware can implement some or all of the following steps:pre-processing, a machine learning wound assessment model, andpost-processing. The machine learning module outputs a conditionalprobability map that is processed by the post-processing module (e.g.,thresholding of probabilities) to generate the results that can then bevisually output to the user in the form of a classified image. As shownin the image output to the user in FIG. 33 , the system can cause animage of the wound to be displayed to the user such that the healingpixels and the non-healing pixels are displayed in different visualrepresentations.

The process of FIG. 33 was applied to a set of DFU images, and thepredicted non-healing region was compared to the true non-healing regionmeasured during a 30-day healing assessment conducted on the DFU. Thiscomparison represents the performance of the algorithm. The methodapplied to perform this comparison was based on the clinical outcome ofthese output images. The database of DFU images contained 28 individualimages of diabetic foot ulcers obtained from 19 subjects. For eachimage, the true area of wound that did not heal after 30 days ofstandard wound care was known. Algorithm training was conducted usingthe leave-one-out cross-validation (LOOCV) procedure. The results ofeach image were computed as one of True Positive, True Negative, FalsePositive, or False Negative. These results were summarized using theperformance metrics described in Table 4 above.

A convolutional neural network was used to generate the conditionalprobability map for each input image. The algorithm includes an inputlayer, convolutional layers, deconvolutional layers, and output layer.The MSI or RGB data is typically input to a convolutional layer. Theconvolutional layer typically consists of a convolution stage (e.g.,affine transformation) whose output is in turn used as input to adetector stage (e.g., nonlinear transformation such as rectified linear[ReLU]), the results of which may undergo further convolutions anddetector stages. These results may be down sampled by a pooling functionor be used directly as the results of the convolutional layer. Theresults of the convolution layer are provided as input to the nextlayer. The deconvolution layers typically begin with a reverse poolinglayer followed by convolution and detector stages. Typically, theselayers are organized in the order of input layer, convolution layers,and then deconvolution layers. This organization is often referred tohaving first the encoder layers followed by the decoder layers. Theoutput layer typically consists of multiple fully connected neuralnetworks applied to each vector across one of the dimensions of thetensor outputted from the previous layer. The aggregation of the resultsfrom these fully connected neural networks is a matrix called theconditional probability map.

Each entry in the conditional probability map represents a region of theoriginal DFU image. This region may be a 1-to-1 mapping with the pixelsin the input MSI image, or an n-to-1 mapping where n is some aggregationof pixels in the original image. The conditional probability values inthis map represent the probability that the tissue in that area of theimage will not respond to standard wound care. The result is asegmentation of the pixels in the original image wherein the predictednon-healing regions are segmented from the predicted healing regions.

The results of a layer within the convolutional neural network can bemodified by information from another source. In this example, clinicaldata from a subject's medical history or treatment plan (e.g., patienthealth metrics or clinical variables as described herein) can be used asthe source of this modification. Thus, the results of the convolutionalneural network can be conditioned on the level of a non-imagingvariable. To do this, feature-wise linear transformation (FiLM) layerscan be incorporated into the network architecture as shown in FIG. 34 .The FiLM layer is a machine learning algorithm trained to learn theparameters of an affine transformation that is applied to one of thelayers in the convolutional neural network. The input to this machinelearning algorithm is a vector of values, in this case the clinicallyrelevant patient medical history in the form of patient health metricvalues or clinical variables. The training of this machine learningalgorithm may be accomplished simultaneously with the training of theconvolutional neural network. One or more FiLM layers with varyinginputs and machine learning algorithms can be applied to various layersof the convolutional neural network.

Input data for the conditional probability mapping includedmultispectral imaging (MSI) data and color photography images (RGBimages). The MSI data consisted of 8 channels of 2D images, where eachof the 8 channels represented the diffuse reflectance of light from thetissue at a specific wavelength filter. The field of view of eachchannel was 15 cm×20 cm with a resolution of 1044 pixels×1408 pixels.The 8 wavelengths included: 420 nm±20 nm, 525 nm±35 nm, 581 nm±20 nm,620 nm±20 nm, 660 nm±20 nm, 726 nm±41 nm, 820 nm±20 nm, and 855 nm±30nm, as illustrated in FIG. 26 . The RGB images included 3 channels of 2Dimages, where each of the 3 channels represented the diffuse reflectanceof light from the tissue at the wavelengths utilized in a traditionalcolor camera sensor. The field of view of each channel was 15 cm×20 cmwith a resolution of 1044 pixels×1408 pixels.

To perform image segmentation on the basis of healing probability, theCNN architecture called SegNet was used. This model was used asdescribed by the original authors to take RGB images as input and outputthe conditional probability map. Additionally, it was modified toutilize the 8-channel MSI images in the input layer. Lastly, the SegNetarchitecture was modified to include a FiLM layer.

To demonstrate that the segmentation of DFU images into healing andnon-healing regions can be accomplished, a variety of deep learningmodels were developed that each utilize different inputs. These modelsused the following two input feature categories: MSI data alone, and RGBimages alone. In addition to varying the input features, a number ofaspects of the algorithm training were varied. Some of these variationsincluded pre-training the model with the PASCAL visual object classes(VOC) data set, pre-training the model with an image database of anothertype of tissue wound, pre-specifying the kernels of the input layer witha filter bank, early stopping, random image augmentations duringalgorithm training, and averaging the results of random imageaugmentations during inferencing to produce a single aggregatedconditional probability map.

The top two performing models from each of the two feature inputcategories were identified to perform better than random chance. Resultsimproved as RGB data was replaced with MSI data. The number ofimage-based errors reduced from 9 to 7. However, it was determined thatboth MSI and RGB methods are feasible for producing a conditionalprobability map for DFU healing potential.

In addition to determining that a SegNet architecture can yielddesirable segmentation accuracy for wound images, it was also determinedthat other types of wound images may be unexpectedly suitable for use intraining systems to segment DFU images or other wound images on thebasis of conditational probability mapping for healing. As describedabove, a SegNet CNN architecture may be suitable for DFU imagesegmentation when pre-trained using DFU image data as training data.However, in some cases a suitably large set of training images may notbe available for certain types of wounds. FIG. 35 illustrates an examplecolor DFU image (A), and four examples of segmentation of the DFU intopredicted healing and non-healing regions by different segmentationalgorithms. In image (A), which was captured on the day of initialassessment, the dashed line indicates the portion of the wound that wasidentified as non-healing in a subsequent assessment four weeks later.In images (B)-(E), each corresponding segmentation algorithm yields asection of predicted non-healing tissue indicated by shading. As shownin image (E), a SegNet algorithm, which was pre-trained using a burnimage database rather than a DFU image database, nevertheless produced ahighly accurate prediction of a non-healing tissue region that closelymatches the contour of the dashed line in image (A) corresponding to anempirically determined non-healing area. In contrast, a naïve Bayeslinear model trained with DFU image data (image (B)), a logisticregression model trained with DFU image data (image (C), and a SegNetpre-trained using PACAL VOC data (image (D)) all showed inferiorresults, with each of images (B)-(D) indicating a much larger andinaccurately shaped area of non-healing tissue.

Example Individual Wavelength Analysis of DFU Images

In further example implementaitons, it has been found that the percentarea reduction (PAR) of a wound at day 30, and/or segmentation in theform of a conditional probability map, can further be performed based onimage data of a single wavelength band, rather than using MSI or RGBimage data. To accomplish this method, a machine learning algorithm wastrained to take features extracted from a single wavelength band imageas input and output a scalar value representing the predicted PAR.

All images were obtained from subjects under an institutional reviewboard (IRB) approved clinical study protocol. The dataset contained 28individual images of diabetic foot ulcers obtained from 17 subjects.Each subject was imaged on their initial visit for treatment of thewounds. Wounds were at least 1.0 cm wide in their longest dimension.Only subjects prescribed standard wound care therapy were included inthe study. To determine the the true PAR after 30 days of treatment, aDFU healing assessment was performed by the clinician during a routinefollow-up visit. In this healing assessment, an image of the wound wascollected and compared to the image taken at day 0 to accuratelyquantify PAR.

Various machine learning algorithms, such as classifier ensembles or thelike, may be used. Two machine learning algorithms for regression wereemployed in this analysis. One algorithm was a bagging ensemble ofdecision tree classifiers (bagged trees), and the second was a randomforest ensemble. All features used for training the machine learningregression models were obtained from the DFU image obtained prior totreatment at the initial visit for the DFU included the study.

Eight grayscale images of each DFU were obtained from unique wavelengthsin the visible and near-infrared spectrum. The field of view of eachimage was approximately 15 cm×20 cm with a resolution of 1044pixels×1408 pixels. The eight unique wavelength were selected using aset of optical band-pass filters with the following wavelength bands:420 nm±20 nm, 525 nm±35 nm, 581 nm±20 nm, 620 nm±20 nm, 660 nm±20 nm,726 nm±41 nm, 820 nm±20 nm, and 855 nm±30 nm, as illustrated in FIG. 26.

Each raw 1044 pixels×1408 pixels image included, for each pixel, areflectance intensity value for the pixel. Quantitative features werecalculated based on the reflectance intensity values, including thefirst and second moments (e.g., mean and standard deviation) of thereflectance intensity values. In addition, the median was also computed.

Following these computations, a set of filters can optionally beindividually applied to the raw image to generate multiple imagetransformations. In one particular example, a total of 512 filters canbe used, each having dimensions 7 pixels×7 pixels or another suitablekernel size. FIG. 36 illustrates an example set of 512 7×7 filterkernels that may be used in an example implementation. This non-limitingexmaple set of filters can be obtained through the training of aconvolutional neural network (CNN) for DFU segmentation. The 512 filtersillustrated in FIG. 36 were obtained from the first set of kernels inthe input layer of the CNN. The “learning” of these filters wasregularized by constraining their weight updates to prevent largedeviations to Gabor filters contained in a filter-bank.

Filters can be applied to the raw image by convolution. From the 512images that result from these filter convolutions, a single 3D matrixmay be constructed with dimensions 512 channels×1044 pixels×1408 pixels.Additional features may then be computed from this 3D matrix. Forexample, in some embodiments the mean, median, and standard deviation ofthe intensity values of the 3D matrix may be computed as furtherfeatures for input into the machine learning algorithm.

In addition to the six features described above (e.g., mean, median, andstandard deviation of pixel values of the raw image and of the 3D matrixconstructed from the application of convolutional filters to the rawimage), additional features and/or linear or non-linear combinations ofsuch features may further be included as desired. For example, theproduct or the ratio of two features could be used as new input featuresto the algorithm. In one example, the product of a mean and a median maybe used as an additional in put feature.

Algorithm training was conducted using the leave-one-outcross-validation (LOOCV) procedure. One DFU was selected for the testset and the remaining DFU images used as the training set. Aftertraining, the model was used to predict the percent area reduction forthe held-out DFU image. Once this was done, the held-out image wasreturned to the full set of DFU images so that this process could berepeated with a different held-out image. LOOCV was repeated until eachDFU image was part of the held-out set once. After accumulating test setresults across every fold of cross-validation, the overall performanceof the model was computed.

The predicted percent area reduction for each DFU image was compared tothe true percent area reduction measured during a 30-day healingassessment conducted on the DFU. The performance of the algorithm wasscored using coefficient of determination (R²). The R² value was used todetermine the utility of each individual wavelength, which is a measureof the proportion of the variance in DFU percent area reduction that wasexplained by the features extracted from the DFU image. The R² value isdefined as:

${R^{2} = {1 - \frac{{\Sigma}_{i}\left( {y_{i} - {f\left( x_{i} \right)}} \right)^{2}}{{\Sigma}_{i}\left( {y_{i} - \overset{\_}{y}} \right)^{2}}}},$

where y_(i) is the true PAR for DFU i, y is the mean PAR across all DFUsin the data set, and f(x_(i)) is the predicted PAR for DFU i. The 95%confidence interval of the R² value was computed from the predictionresults of the algorithm trained on each feature set. The 95% CI wascomputed using the following equation:

R²+2*SE_(R) ₂ ,

where

${SE}_{R^{2}} = \sqrt{\frac{4{R^{2}\left( {1 - R^{2}} \right)}^{2}\left( {n - k - 1} \right)^{2}}{\left( {n^{2} - 1} \right)\left( {n + 3} \right)}}$

In this equation n is the total number DFU images in the data set and kis the total number of predictors in the model.

The goal was to determine that each of the eight individual wavelengthscould be used independently in a regression model to achieve resultsthat were significantly better than random chance. To determine if afeature set could provide an improvement over random chance, featuresets were identified wherein zero was not contained within the 95% CI ofR² for the algorithm trained on that feature set. To do this, eightseparate experiments were conducted wherein models were trained with thefollowing six original features: the mean, median, and standarddeviation of the raw image; and the mean, median, and standard deviationof the 3D matrix generated from raw image transformations by applicationof the convolutional filters. The random forest and bagged trees modelswere trained. Results were reported for the algorithm with superiorperformance in cross-validation. The results of these eight models werereviewed to determine whether the lower-bound 95% CI was above zero. Ifnot, the additional features generated by non-linear combinations of thesix original features were employed.

Using the six original features, seven of the eight wavelengths examinedcould be used to generate regression models that explained a significantamount of the variance in percent area reduction from the DFU dataset.In order of most effective to least effective, the seven wavelengthswere: 660 nm; 620 nm; 726 nm; 855 nm; 525 nm; 581 nm; and 420 nm. Thefinal wavelength, 820 nm, was found to be significant if the product ofmean and median of the 3D matrix was included as an additional feature.Results of these trials are summarized in Table 5.

TABLE 5 Results of regression models developed for the eight uniquewavelength images Lower Wavelength R² 95% CI Input Features AlgorithmAlgorithm Parameters 660 0.410 0.210 Original six Random Forestn_estimators = 1 620 0.340 0.137 Original six Random Forest n_estimators= 1 726 0.270 0.060 Original six Bagged Trees n_estimators = 1max_features = 5 max_samples = 19 855 0.230 0.026 Original six BaggedTrees n_estimators = 1 max_features = 5 max_samples = 15 525 0.210 0.010Original six Bagged Trees n_estimators = 1 max_features = 5 max_samples= 25 581 0.200 0.002 Original six Bagged Trees n_estimators = 2max_features = 5 max_samples = 24 420 0.190 0.003 Original six RandomForest n_estimators = 2 820 0.183 0.007 Original six, and the BaggedTrees n_estimators = 2 product of max_features = 7 Mean(transformed)max_samples = 5 with Median(transformed)

Accordingly, it has been shown that the imaging and analysis systems andmethods described herein may be able to accurately generate one or morepredicted healing parameters based on even a single wavelength bandimage. In some embodiments, use of a single wavelength band may befacilitated the calculation of one or more aggregate quantitativefeatures from the image, such as a mean, median, or standard deviationof raw image data and/or of a set of images or 3D matrix generated byapplication of one or more filters to the raw image data.

Example Wound Image Segmentation Systems and Methods

As described above, spectral images including reflectance data at anindividual wavelength or a plurality of wavelengths can be analyzedusing the machine learning techniques described herein, to reliablypredict parameters associated with wound healing, such as overall woundhealing (e.g., percent area reduction) and/or healing associated withportions of a wound (e.g., a healing probability associated with anindividual pixel or subset of pixels of a wound image). Moreover, someof the methods disclosed herein predict wound healing parameters basedat least in part on aggregate quantitative features, for example,statistical quanitities such as means, standard deviations, medianvalues, or the like, calculated based on a subset of pixels of a woundimage that are determined to be the “wound pixels,” or the pixels thatcorrespond to the wound tissue region rather than callus, normal skin,background, or other non-wound tissue regions. Accordingly, in order toimprove or optimize the accuracy of such predictions based on a set ofwound pixels, it is preferable to accurately select the subset of woundpixels in an image of a wound.

Conventionally, segmentation of an image such as an image of a DFU intowound pixels and non-wound pixels has been performed manually, forexample, by a doctor or other clinician who examines the image andselects the set of wound pixels based on the image. However, such manualsegmentation may be time consuming, inefficient, and potentially proneto human error. For example, the formulas used to compute area andvolume lack the accuracy and precision required to measure the convexshape of wounds. In addition, identifying the true boundaries of thewound and classification of tissues within the wound, such as epithelialgrowth, requires a high level of competency. Since changes in woundmeasurements are often the critical information used to determinetreatment efficacy, errors in the initial wound measurements can resultin incorrect treatment determinations.

To this end, systems and methods of the present technology are suitablefor automated detection of wound margins and identification of tissuetypes in the wound area. In some embodiments, the systems and methods ofthe present technology can be configured for automated segmentation ofwound images into at least wound pixels and non-wound pixels, such thatany aggregate quantitiative features calculated based on the subset ofwound pixels achieve a desirable level of accuracy. Moreover, it may bedesirable to implement systems or methods capable of segmenting a woundimage into wound and non-wound pixels, and/or into one or moresub-classes of wound or non-wound pixels, without necessarily furthergenerating predicted healing parameters.

A dataset of diabetic foot ulcer images may be developed using colorphotographs of wounds. Various color camera systems can be used in theacquisition of this data. In one example implementation, 349 totalimages were used. A trained physician or other clinician may use asoftware program to identify and label the wound, callus, normal skin,background, and/or any other types of pixel categories in each woundimage. The resulting labeled images, known as ground truth maks, mayinclude a number of colors corresponding to the number of labeledcategories in the image. FIG. 37 illustrates an example image of a DFU(left), and corresponding ground truth mask (right). The example groundtruth mask of FIG. 37 includes a purple region corresponding tobackground pixels, a yellow region corresponding to callus pixels, and acyan region corresponding to wound pixels.

Based on a set of ground truth images, a convolutional neural network(CNN) can be used for the automated segmentation of these tissuecategories. In some embodiments, the algorithm structure can be ashallow U-net with a plurality of convolutional layers. In one exampleimplementation, desirable segmentation outcomes were achieved with 31convolutional layers. However, many other algorithms for imagesegmentation could be applied to achieve the desired output.

In the example segmentation implementation, the DFU image database wasrandomly split into three sets such that 269 training set images wereused for algorithm training, 40 test set images for hyperparameterselection, and 40 validation set images for validation. The algorithmwas trained with gradient descent and the accuracy of the test setimages was monitored. The algorithm training was stopped when the testset accuracy was maximized. The results of this algorithm were thendetermined using the validation set.

Results from the U-net algorithm for each image in the validation setwere compared to their corresponding ground truth mask. This comparisonwas done on a pixel-by-pixel basis. Within each of the three tissuetypes this comparison was summarized using the following categories. ATrue Positive (TP) category included the total number of pixels forwhich the tissue type of interest was present at a pixel in the groundtruth mask, and the model predicted the tissue type was present at thispixel. A True Negative (TN) category included the total number of pixelsfor which the tissue type of interest was not present at a pixel in theground truth mask, and the model predicted the tissue type was notpresent at this pixel. A False Positive (FP) category included he totalnumber of pixels for which the tissue type of interest was not presentat a pixel in the ground truth mask, and the model predicted the tissuetype was present at this pixel. A False Negative (FN) category includedthe total number of pixels for which the tissue type of interest waspresent at a pixel in the ground truth mask, and the model predicted thetissue type was not present at this pixel. These results were summarizedusing the following metrics:

Accuracy:

${{Acc_{model}} = {{\frac{1}{N}{{\sum}_{i = 1}^{N}\left\lbrack {{TP_{wound}} + {TP_{callus}} + {TP_{u{ninjured}}}} \right\rbrack}} + \text{ }\left\lbrack {{TN}_{wound} + {TN_{callus}} + {TN_{u{ninjured}}}} \right\rbrack}},$

where N is the total number of pixels in the validation set.

Average dice score:

${{AveDSC_{model}} = {\frac{1}{C}{\Sigma}_{j = 1}^{C}\frac{2TP_{j}}{{2TP_{j}} + {FP_{j}} + {FN_{j}}}}},$

where C represents the three tissue types.

Average intersection over union (IOU):

${{AveDSC_{model}} = {\frac{1}{C}{\Sigma}_{j = 1}^{C}\frac{TP_{j}}{{TP_{j}} + {FP_{j}} + {FN_{j}}}}},$

where C represents the three tissue types.

In some embodiments, algorithm training may be conducted over aplurality of epochs, and an intermediate number of epochs may bedetermined at which accuracy is optimized. In the example implementationdescribed herein, algorithm training for image segmentation wasconducted over 80 epochs. As training was monitored, it was determinedthat epoch 73 achieved the best accuracy for test dataset.

The performance of the U-net segmentation algorithm was computed withthe accuracy being better than random chance. U-net also outperformedall three possible naïve approaches, where a naïve classifier is used toalways predict one tissue class. Regardless of the potential overfittingissue, the model performance on the validation set was able todemonstrate feasibility based on these summary metrics.

FIG. 38 illustrates three example results of wound image segmentationusing the U-net segmentation algorithm in combination with the methodsdescribed herein. For each of the three example DFU images in the rightcolumn, the U-net segmentation algorithm, trained as described herein,generated the automated image segmentation outputs shown in the middlecolumn. The manually generated ground truth masks corresponding to eachDFU image are shown in the left column of FIG. 38 , visuallyillustrating the high segmentation accuracy that can be obtained usingthe methods described herein.

Terminology

All of the methods and tasks described herein may be performed and fullyautomated by a computer system. The computer system may, in some cases,include multiple distinct computers or computing devices (e.g., physicalservers, workstations, storage arrays, cloud computing resources, etc.)that communicate and interoperate over a network to perform thedescribed functions. Each such computing device typically includes aprocessor (or multiple processors) that executes program instructions ormodules stored in a memory or other non-transitory computer-readablestorage medium or device (e.g., solid state storage devices, diskdrives, etc.). The various functions disclosed herein may be embodied insuch program instructions, or may be implemented in application-specificcircuitry (e.g., ASICs or FPGAs) of the computer system. Where thecomputer system includes multiple computing devices, these devices may,but need not, be co-located. The results of the disclosed methods andtasks may be persistently stored by transforming physical storagedevices, such as solid-state memory chips or magnetic disks, into adifferent state. In some embodiments, the computer system may be acloud-based computing system whose processing resources are shared bymultiple distinct business entities or other users.

The disclosed processes may begin in response to an event, such as on apredetermined or dynamically determined schedule, on demand wheninitiated by a user or system administer, or in response to some otherevent. When the process is initiated, a set of executable programinstructions stored on one or more non-transitory computer-readablemedia (e.g., hard drive, flash memory, removable media, etc.) may beloaded into memory (e.g., RAM) of a server or other computing device.The executable instructions may then be executed by a hardware-basedcomputer processor of the computing device. In some embodiments, theprocess or portions thereof may be implemented on multiple computingdevices and/or multiple processors, serially or in parallel.

Depending on the embodiment, certain acts, events, or functions of anyof the processes or algorithms described herein can be performed in adifferent sequence, can be added, merged, or left out altogether (e.g.,not all described operations or events are necessary for the practice ofthe algorithm). Moreover, in certain embodiments, operations or eventscan be performed concurrently, e.g., through multi-threaded processing,interrupt processing, or multiple processors or processor cores or onother parallel architectures, rather than sequentially.

The various illustrative logical blocks, modules, routines, andalgorithm steps described in connection with the embodiments disclosedherein can be implemented as electronic hardware (e.g., ASICs or FPGAdevices), computer software that runs on computer hardware, orcombinations of both. Moreover, the various illustrative logical blocksand modules described in connection with the embodiments disclosedherein can be implemented or performed by a machine, such as a processordevice, a digital signal processor (“DSP”), an application specificintegrated circuit (“ASIC”), a field programmable gate array (“FPGA”) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A processor device can be amicroprocessor, but in the alternative, the processor device can be acontroller, microcontroller, or state machine, combinations of the same,or the like. A processor device can include electrical circuitryconfigured to process computer-executable instructions. In anotherembodiment, a processor device includes an FPGA or other programmabledevice that performs logic operations without processingcomputer-executable instructions. A processor device can also beimplemented as a combination of computing devices, e.g., a combinationof a DSP and a microprocessor, a plurality of microprocessors, one ormore microprocessors in conjunction with a DSP core, or any other suchconfiguration. Although described herein primarily with respect todigital technology, a processor device may also include primarily analogcomponents. For example, some or all of the rendering techniquesdescribed herein may be implemented in analog circuitry or mixed analogand digital circuitry. A computing environment can include any type ofcomputer system, including, but not limited to, a computer system basedon a microprocessor, a mainframe computer, a digital signal processor, aportable computing device, a device controller, or a computationalengine within an appliance, to name a few.

The elements of a method, process, routine, or algorithm described inconnection with the embodiments disclosed herein can be embodieddirectly in hardware, in a software module executed by a processordevice, or in a combination of the two. A software module can reside inRAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, hard disk, a removable disk, a CD-ROM, or any other form of anon-transitory computer-readable storage medium. An exemplary storagemedium can be coupled to the processor device such that the processordevice can read information from, and write information to, the storagemedium. In the alternative, the storage medium can be integral to theprocessor device. The processor device and the storage medium can residein an ASIC. The ASIC can reside in a user terminal. In the alternative,the processor device and the storage medium can reside as discretecomponents in a user terminal.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements or steps.Thus, such conditional language is not generally intended to imply thatfeatures, elements or steps are in any way required for one or moreembodiments or that one or more embodiments necessarily include logicfor deciding, with or without other input or prompting, whether thesefeatures, elements or steps are included or are to be performed in anyparticular embodiment. The terms “comprising,” “including,” “having,”and the like are synonymous and are used inclusively, in an open-endedfashion, and do not exclude additional elements, features, acts,operations, and so forth. Also, the term “or” is used in its inclusivesense (and not in its exclusive sense) so that when used, for example,to connect a list of elements, the term “or” means one, some, or all ofthe elements in the list.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is otherwise understood with thecontext as used in general to present that an item, term, etc., may beeither X, Y, or Z, or any combination thereof (e.g., X, Y, or Z). Thus,such disjunctive language is not generally intended to, and should not,imply that certain embodiments require at least one of X, at least oneof Y, and at least one of Z to each be present.

While the above detailed description has shown, described, and pointedout novel features as applied to various embodiments, it can beunderstood that various omissions, substitutions, and changes in theform and details of the devices or algorithms illustrated can be madewithout departing from the scope of the disclosure. As can berecognized, certain embodiments described herein can be embodied withina form that does not provide all of the features and benefits set forthherein, as some features can be used or practiced separately fromothers. All changes which come within the meaning and range ofequivalency of the claims are to be embraced within their scope.

What is claimed is:
 1. A computer-implemented method of assessing orpredicting wound healing comprising: receiving, from at least one lightdetection element, a signal representing light of at least a firstwavelength reflected from a tissue region comprising a wound or portionthereof; generating, based on the signal, an image having a plurality ofpixels depicting the tissue region; determining, based on the signal, areflectance intensity value at the first wavelength for each pixel of atleast a subset of the plurality of pixels; determining one or morequantitative features of the subset of the plurality of pixels based onthe reflectance intensity values of each pixel of the subset; andgenerating, using one or more machine learning algorithms, at least onescalar value based on the one or more quantitative features of thesubset of the plurality of pixels, the at least one scalar valuecorresponding to a predicted amount of healing of the wound or portionthereof over a predetermined time interval following generation of theimage.
 2. The computer-implemented method of claim 1, further comprisingdetermining the predicted amount of healing of the wound or portionthereof over the predetermined time interval.
 3. Thecomputer-implemented method of claim 1, wherein the predicted amount ofhealing is a predicted percent area reduction of the wound or portionthereof.
 4. The computer-implemented method of claim 1, wherein thepredetermined time interval is 30 days.
 5. The computer-implementedmethod of claim 1, further comprising identifying at least one patienthealth metric value corresponding to a patient having the tissue region,and wherein the at least one scalar value is generated based on the oneor more quantitative features of the subset of the plurality of pixelsand on the at least one patient health metric value.
 6. Thecomputer-implemented method of claim 5, wherein the at least one patienthealth metric value comprises at least one variable selected fromdemographic variables, compliance variables, endocrine variables,cardiovascular variables, musculoskeletal variables, nutritionvariables, infectious disease variables, renal variables, obstetrics orgynecology variables, drug use variables, other disease variables, orlaboratory values.
 7. The computer-implemented method of claim 5,wherein the at least one patient health metric value comprises at leastone feature selected from the group consisting of an age of the patient,a level of chronic kidney disease of the patient, a length of the woundor portion thereof on a day when the image is generated, and a width ofthe wound or portion thereof on the day when the image is generated. 8.The computer-implemented method of claim 1, wherein the first wavelengthis within the range of 620 nm±20 nm, 660 nm±20 nm, or 420 nm±20 nm, andwherein the one or more machine learning algorithms comprise a randomforest ensemble.
 9. The computer-implemented method of claim 1, whereinthe first wavelength is within the range of 726 nm±41 nm, 855 nm±30 nm,525 nm±35 nm, 581 nm±20 nm, or 820 nm±20 nm, and wherein the one or moremachine learning algorithms comprise an ensemble of classifiers.
 10. Thecomputer-implemented method of claim 1, further comprising:automatically segmenting the plurality of pixels of the image into woundpixels and non-wound pixels; and selecting the subset of the pluralityof pixels to comprise the wound pixels.
 11. The computer-implementedmethod of claim 10, wherein the the plurality of pixels areautomatically segmented using a segmentation algorithm comprising atleast one of a U-Net comprising a plurality of convolutional layers anda SegNet comprising a plurality of convolutional layers.
 12. Thecomputer-implemented method of claim 1, wherein the one or morequantitative features of the subset of the plurality of pixels areselected from the group consisting of a mean of the reflectanceintensity values of the pixels of the subset, a standard deviation ofthe reflectance intensity values of the pixels of the subset, and amedian reflectance intensity value of the pixels of the subset.
 13. Thecomputer-implemented method of claim 1, further comprising: individuallyapplying a plurality of filter kernels to the image by convolution togenerate a plurality of image transformations; constructing a 3D matrixfrom the plurality of image transformations; and determining one or morequantitative features of the 3D matrix, wherein the at least one scalarvalue is generated based on the one or more quantitative features of thesubset of the plurality of pixels and on the one or more quantitativefeatures of the 3D matrix.
 14. The computer-implemented method of claim13, wherein the one or more quantitative features of the 3D matrix areselected from the group consisting of a mean of the values of the 3Dmatrix, a standard deviation of the values of the 3D matrix, a medianvalue of the 3D matrix, and a product of the mean and the median of the3D matrix.
 15. The computer-implemented method of claim 14, wherein theat least one scalar value is generated based on the mean of thereflectance intensity values of the pixels of the subset, the standarddeviation of the reflectance intensity values of the pixels of thesubset, the median reflectance intensity value of the pixels of thesubset, the mean of the values of the 3D matrix, the standard deviationof the values of the 3D matrix, and the median value of the 3D matrix.16. The computer-implemented method of claim 1, further comprising:receiving a second signal from the at least one light detection element,the second signal representing light of a second wavelength reflectedfrom the tissue region; determining, based on the second signal, areflectance intensity value at the second wavelength for each pixel ofat least the subset of the plurality of pixels; and determining one ormore additional quantitative features of the subset of the plurality ofpixels based on the reflectance intensity values of each pixel at thesecond wavelength; wherein the at least one scalar value is generatedbased at least in part on the one or more additional quantitativefeatures of the subset of the plurality of pixels.
 17. Thecomputer-implemented method of claim 1, wherein the at least one scalarvalue is generated on a same day as a beginning of the predeterminedtime interval.
 18. The computer-implemented method of claim 1, furthercomprising: measuring one or more dimensions of the wound or a portionthereof after the predetermined time interval has elapsed following thedetermination of the predicted amount of healing of the wound or portionthereof; determining an actual amount of healing of the wound or portionthereof over the predetermined time interval; and updating at least onemachine learning algorithm of the one or more machine learningalgorithms by providing at least the image and the actual amount ofhealing of the wound or portion thereof as training data.
 19. Thecomputer-implemented method of claim 1, further comprising selecting,prior to an end of the predetermined time interval, between a standardwound care therapy and an advanced wound care therapy based at least inpart on the predicted amount of healing of the wound or portion thereof.20. The computer-implemented method of claim 19, wherein selectingbetween the standard wound care therapy and the advanced wound caretherapy comprises: when the predicted amount of healing indicates thatthe wound or portion thereof will heal or close by greater than 50% in30 days, indicating or applying one or more standard therapies selectedfrom improving nutritional status, debridement to remove devitalizedtissue, maintenance of granulation tissue with a dressing, therapy toaddress any infection that may be present, addressing a deficiency invascular perfusion to an extremity comprising the wound or portionthereof, offloading of pressure from the wound or portion thereof, orglucose regulation; and when the predicted amount of healing indicatesthat the wound or portion thereof will not heal or close by greater than50% in 30 days, indicating or applying one or more advanced caretherapies selected from the group consisting of hyperbaric oxygentherapy, negative-pressure wound therapy, bioengineered skinsubstitutes, synthetic growth factors, extracellular matrix proteins,matrix metalloproteinase modulators, and electrical stimulation therapy.